Rethinking key-value store for parallel I/O optimization

Anthony Kougkas, Hassan Eslami, Xian He Sun, Rajeev Thakur, William Gropp

Research output: Contribution to journalArticlepeer-review

Abstract

Key-value stores are being widely used as the storage system for large-scale internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems are the dominant storage solution. In this study, we examine the architecture differences and performance characteristics of parallel file systems and key-value stores. We propose using key-value stores to optimize overall Input/Output (I/O) performance, especially for workloads that parallel file systems cannot handle well, such as the cases with intense data synchronization or heavy metadata operations. We conducted experiments with several synthetic benchmarks, an I/O benchmark, and a real application. We modeled the performance of these two systems using collected data from our experiments, and we provide a predictive method to identify which system offers better I/O performance given a specific workload. The results show that we can optimize the I/O performance in HPC systems by utilizing key-value stores.

Original languageEnglish (US)
Pages (from-to)335-356
Number of pages22
JournalInternational Journal of High Performance Computing Applications
Volume31
Issue number4
DOIs
StatePublished - Jul 1 2017

Keywords

  • Hyperdex
  • I/O performance
  • OrangeFS
  • key-value store
  • parallel I/O optimization
  • performance evaluation
  • prediction model

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'Rethinking key-value store for parallel I/O optimization'. Together they form a unique fingerprint.

Cite this