Trading memory for disk: Using parallel access to fast InfiniBand disk arrays for large computational chemistry applications

Troy Benjegerdes, Brett Bode, Kyle Schochenmaier

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a novel approach for using high performance network attached parallel storage for out-of-core computation. Our approach utilizes many parallel disks and storage controllers with a near 1:1 ratio of compute nodes to storage servers. This, when combined with 30 Gigabit 12X InfiniBand interconnects, allows remote storage subsystem bandwidth to reach the same performance level as local memory-cache file I/O performance. This combination allows computational chemistry application problem sizes which require more than 100GB of intermediate data to store this data on disk without the disk I/O subsystem becoming the limiting factor. With sequential storage access speeds on the same order of magnitude as main memory performance, this allows out-of-core computation to become practical due to disk storage size being several orders of magnitude cheaper per GB than main memory storage.

Original languageEnglish (US)
Title of host publicationProceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06
DOIs
StatePublished - Dec 1 2006
Externally publishedYes

Publication series

NameProceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06

ASJC Scopus subject areas

  • Computer Science(all)

Fingerprint Dive into the research topics of 'Trading memory for disk: Using parallel access to fast InfiniBand disk arrays for large computational chemistry applications'. Together they form a unique fingerprint.

Cite this