TY - GEN
T1 - Rethinking key-value store for parallel I/O optimization
AU - Yin, Yanlong
AU - Kougkas, Antonios
AU - Feng, Kun
AU - Eslami, Hassan
AU - Lu, Yin
AU - Sun, Xian He
AU - Thakur, Rajeev
AU - Gropp, William
N1 - Publisher Copyright:
© 2014 IEEE.
PY - 2014/4/2
Y1 - 2014/4/2
N2 - Key-Value Stores (KVStore) are being widely used as the storage system for large-scale Internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems (PFS) are the dominant storage systems. In this study, we carefully examine the architecture difference and performance characteristics of PFS and KVStore. We propose that it is valuable to utilize KVStore to optimize the overall I/O performance, especially for the workloads that PFS cannot handle well, such as the cases with hurtful data synchronization or heavy metadata operations. To verify this proposal, we conducted comprehensive experiments with several synthetic benchmarks, an I/O benchmark, and a real application. The results show that our proposal is promising.
AB - Key-Value Stores (KVStore) are being widely used as the storage system for large-scale Internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems (PFS) are the dominant storage systems. In this study, we carefully examine the architecture difference and performance characteristics of PFS and KVStore. We propose that it is valuable to utilize KVStore to optimize the overall I/O performance, especially for the workloads that PFS cannot handle well, such as the cases with hurtful data synchronization or heavy metadata operations. To verify this proposal, we conducted comprehensive experiments with several synthetic benchmarks, an I/O benchmark, and a real application. The results show that our proposal is promising.
UR - http://www.scopus.com/inward/record.url?scp=84946531241&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84946531241&partnerID=8YFLogxK
U2 - 10.1109/DISCS.2014.11
DO - 10.1109/DISCS.2014.11
M3 - Conference contribution
AN - SCOPUS:84946531241
T3 - Proceedings of DISCS 2014: The 2014 International Workshop on Data-Intensive Scalable Computing Systems - Held in Conjuction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis
SP - 33
EP - 40
BT - Proceedings of DISCS 2014
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2014 International Workshop on Data-Intensive Scalable Computing Systems, DISCS 2014 - Held in Conjuction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2014
Y2 - 16 November 2014
ER -