Rethinking key-value store for parallel I/O optimization

Yanlong Yin, Antonios Kougkas, Kun Feng, Hassan Eslami, Yin Lu, Xian He Sun, Rajeev Thakur, William Gropp

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Key-Value Stores (KVStore) are being widely used as the storage system for large-scale Internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems (PFS) are the dominant storage systems. In this study, we carefully examine the architecture difference and performance characteristics of PFS and KVStore. We propose that it is valuable to utilize KVStore to optimize the overall I/O performance, especially for the workloads that PFS cannot handle well, such as the cases with hurtful data synchronization or heavy metadata operations. To verify this proposal, we conducted comprehensive experiments with several synthetic benchmarks, an I/O benchmark, and a real application. The results show that our proposal is promising.

Original languageEnglish (US)
Title of host publicationProceedings of DISCS 2014
Subtitle of host publicationThe 2014 International Workshop on Data-Intensive Scalable Computing Systems - Held in Conjuction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages33-40
Number of pages8
ISBN (Electronic)9781479970384
DOIs
StatePublished - Apr 2 2014
Event2014 International Workshop on Data-Intensive Scalable Computing Systems, DISCS 2014 - Held in Conjuction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2014 - New Orleans, United States
Duration: Nov 16 2014 → …

Publication series

NameProceedings of DISCS 2014: The 2014 International Workshop on Data-Intensive Scalable Computing Systems - Held in Conjuction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis

Other

Other2014 International Workshop on Data-Intensive Scalable Computing Systems, DISCS 2014 - Held in Conjuction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2014
Country/TerritoryUnited States
CityNew Orleans
Period11/16/14 → …

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'Rethinking key-value store for parallel I/O optimization'. Together they form a unique fingerprint.

Cite this