Network-aware data caching and prefetching for Cloud-hosted metadata retrieval

Bing Zhang, Brandon Ross, Sanatkumar Tripathi, Sonali Batra, Tevfik Kosar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the overwhelming emergence of data-intensive applications in the Cloud, the wide-area transfer of metadata and other descriptive information about remote data is critically important for searching, indexing, and enumerating remote file system hierarchies, as well as for purposes of data transfer estimation and reservation. In this paper, we present a highly efficient network-aware caching and prefetching mechanism tailored to reduce metadata access latency and improve responsiveness in wide-area data transfers. To improve the maximum requests per second (RPS) handled by the system, we designed and implemented a network-aware prefetching service using dynamically provisioned parallel TCP streams. To improve the performance of accessing local metadata, we designed and implemented a non-blocking concurrent in-memory cache to handle unexpected bursts of requests. We have implemented the proposed mechanisms in the Directory Listing Service (DLS) system-a Cloud-hosted metadata retrieval, caching, and prefetching system, and have evaluated its performance on Amazon EC2 and XSEDE.

Original languageEnglish (US)
Title of host publicationProc. of NDM 2013
Subtitle of host publication3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis
DOIs
StatePublished - 2013
Event3rd International Workshop on Network-Aware Data Management, NDM 2013 - Held in Conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013 - Denver, CO, United States
Duration: Nov 17 2013Nov 17 2013

Publication series

NameProc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference3rd International Workshop on Network-Aware Data Management, NDM 2013 - Held in Conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013
CountryUnited States
CityDenver, CO
Period11/17/1311/17/13

Keywords

  • Caching
  • Metadata retrieval
  • Prefetching
  • Software as a service (SaaS)
  • Wide-area transfers

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications

Fingerprint Dive into the research topics of 'Network-aware data caching and prefetching for Cloud-hosted metadata retrieval'. Together they form a unique fingerprint.

Cite this