Scalable Reed-Solomon-based reliable local storage for HPC applications on IaaS clouds

Leonardo Bautista Gomez, Bogdan Nicolae, Naoya Maruyama, Franck Cappello, Satoshi Matsuoka

Research output: Chapter in Book/Report/Conference proceedingConference contribution


With increasing interest among mainstream users to run HPC applications, Infrastructure-as-a-Service (IaaS) cloud computing platforms represent a viable alternative to the acquisition and maintenance of expensive hardware, often out of the financial capabilities of such users. Also, one of the critical needs of HPC applications is an efficient, scalable and persistent storage. Unfortunately, storage options proposed by cloud providers are not standardized and typically use a different access model. In this context, the local disks on the compute nodes can be used to save large data sets such as the data generated by Checkpoint-Restart (CR). This local storage offers high throughput and scalability but it needs to be combined with persistency techniques, such as block replication or erasure codes. One of the main challenges that such techniques face is to minimize the overhead of performance and I/O resource utilization (i.e., storage space and bandwidth), while at the same time guaranteeing high reliability of the saved data. This paper introduces a novel persistency technique that leverages Reed-Solomon (RS) encoding to save data in a reliable fashion. Compared to traditional approaches that rely on block replication, we demonstrate about 50% higher throughput while reducing network bandwidth and storage utilization by a factor of 2 for the same targeted reliability level. This is achieved both by modeling and real life experimentation on hundreds of nodes.

Original languageEnglish (US)
Title of host publicationParallel Processing - 18th International Conference, Euro-Par 2012, Proceedings
Number of pages12
StatePublished - 2012
Event18th International Conference on Parallel Processing, Euro-Par 2012 - Rhodes Island, Greece
Duration: Aug 27 2012Aug 31 2012

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume7484 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Other18th International Conference on Parallel Processing, Euro-Par 2012
CityRhodes Island

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'Scalable Reed-Solomon-based reliable local storage for HPC applications on IaaS clouds'. Together they form a unique fingerprint.

Cite this