Network-aware data caching and prefetching for Cloud-hosted metadata retrieval

Bing Zhang, Brandon Ross, Sanatkumar Tripathi, Sonali Batra, Tevfik Kosar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the overwhelming emergence of data-intensive applications in the Cloud, the wide-area transfer of metadata and other descriptive information about remote data is critically important for searching, indexing, and enumerating remote file system hierarchies, as well as for purposes of data transfer estimation and reservation. In this paper, we present a highly efficient network-aware caching and prefetching mechanism tailored to reduce metadata access latency and improve responsiveness in wide-area data transfers. To improve the maximum requests per second (RPS) handled by the system, we designed and implemented a network-aware prefetching service using dynamically provisioned parallel TCP streams. To improve the performance of accessing local metadata, we designed and implemented a non-blocking concurrent in-memory cache to handle unexpected bursts of requests. We have implemented the proposed mechanisms in the Directory Listing Service (DLS) system-a Cloud-hosted metadata retrieval, caching, and prefetching system, and have evaluated its performance on Amazon EC2 and XSEDE.

Original languageEnglish (US)
Title of host publicationProc. of NDM 2013
Subtitle of host publication3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis
DOIs
StatePublished - Dec 1 2013
Event3rd International Workshop on Network-Aware Data Management, NDM 2013 - Held in Conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013 - Denver, CO, United States
Duration: Nov 17 2013Nov 17 2013

Publication series

NameProc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference3rd International Workshop on Network-Aware Data Management, NDM 2013 - Held in Conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013
CountryUnited States
CityDenver, CO
Period11/17/1311/17/13

Fingerprint

Metadata
Data transfer
Cache memory

Keywords

  • Caching
  • Metadata retrieval
  • Prefetching
  • Software as a service (SaaS)
  • Wide-area transfers

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications

Cite this

Zhang, B., Ross, B., Tripathi, S., Batra, S., & Kosar, T. (2013). Network-aware data caching and prefetching for Cloud-hosted metadata retrieval. In Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis [a4] (Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis). https://doi.org/10.1145/2534695.2534700

Network-aware data caching and prefetching for Cloud-hosted metadata retrieval. / Zhang, Bing; Ross, Brandon; Tripathi, Sanatkumar; Batra, Sonali; Kosar, Tevfik.

Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis. 2013. a4 (Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Zhang, B, Ross, B, Tripathi, S, Batra, S & Kosar, T 2013, Network-aware data caching and prefetching for Cloud-hosted metadata retrieval. in Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis., a4, Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis, 3rd International Workshop on Network-Aware Data Management, NDM 2013 - Held in Conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013, Denver, CO, United States, 11/17/13. https://doi.org/10.1145/2534695.2534700
Zhang B, Ross B, Tripathi S, Batra S, Kosar T. Network-aware data caching and prefetching for Cloud-hosted metadata retrieval. In Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis. 2013. a4. (Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis). https://doi.org/10.1145/2534695.2534700
Zhang, Bing ; Ross, Brandon ; Tripathi, Sanatkumar ; Batra, Sonali ; Kosar, Tevfik. / Network-aware data caching and prefetching for Cloud-hosted metadata retrieval. Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis. 2013. (Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis).
@inproceedings{1a0e7c4fed5b4152baced0da65d0ad2d,
title = "Network-aware data caching and prefetching for Cloud-hosted metadata retrieval",
abstract = "With the overwhelming emergence of data-intensive applications in the Cloud, the wide-area transfer of metadata and other descriptive information about remote data is critically important for searching, indexing, and enumerating remote file system hierarchies, as well as for purposes of data transfer estimation and reservation. In this paper, we present a highly efficient network-aware caching and prefetching mechanism tailored to reduce metadata access latency and improve responsiveness in wide-area data transfers. To improve the maximum requests per second (RPS) handled by the system, we designed and implemented a network-aware prefetching service using dynamically provisioned parallel TCP streams. To improve the performance of accessing local metadata, we designed and implemented a non-blocking concurrent in-memory cache to handle unexpected bursts of requests. We have implemented the proposed mechanisms in the Directory Listing Service (DLS) system-a Cloud-hosted metadata retrieval, caching, and prefetching system, and have evaluated its performance on Amazon EC2 and XSEDE.",
keywords = "Caching, Metadata retrieval, Prefetching, Software as a service (SaaS), Wide-area transfers",
author = "Bing Zhang and Brandon Ross and Sanatkumar Tripathi and Sonali Batra and Tevfik Kosar",
year = "2013",
month = "12",
day = "1",
doi = "10.1145/2534695.2534700",
language = "English (US)",
isbn = "9781450325226",
series = "Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis",
booktitle = "Proc. of NDM 2013",

}

TY - GEN

T1 - Network-aware data caching and prefetching for Cloud-hosted metadata retrieval

AU - Zhang, Bing

AU - Ross, Brandon

AU - Tripathi, Sanatkumar

AU - Batra, Sonali

AU - Kosar, Tevfik

PY - 2013/12/1

Y1 - 2013/12/1

N2 - With the overwhelming emergence of data-intensive applications in the Cloud, the wide-area transfer of metadata and other descriptive information about remote data is critically important for searching, indexing, and enumerating remote file system hierarchies, as well as for purposes of data transfer estimation and reservation. In this paper, we present a highly efficient network-aware caching and prefetching mechanism tailored to reduce metadata access latency and improve responsiveness in wide-area data transfers. To improve the maximum requests per second (RPS) handled by the system, we designed and implemented a network-aware prefetching service using dynamically provisioned parallel TCP streams. To improve the performance of accessing local metadata, we designed and implemented a non-blocking concurrent in-memory cache to handle unexpected bursts of requests. We have implemented the proposed mechanisms in the Directory Listing Service (DLS) system-a Cloud-hosted metadata retrieval, caching, and prefetching system, and have evaluated its performance on Amazon EC2 and XSEDE.

AB - With the overwhelming emergence of data-intensive applications in the Cloud, the wide-area transfer of metadata and other descriptive information about remote data is critically important for searching, indexing, and enumerating remote file system hierarchies, as well as for purposes of data transfer estimation and reservation. In this paper, we present a highly efficient network-aware caching and prefetching mechanism tailored to reduce metadata access latency and improve responsiveness in wide-area data transfers. To improve the maximum requests per second (RPS) handled by the system, we designed and implemented a network-aware prefetching service using dynamically provisioned parallel TCP streams. To improve the performance of accessing local metadata, we designed and implemented a non-blocking concurrent in-memory cache to handle unexpected bursts of requests. We have implemented the proposed mechanisms in the Directory Listing Service (DLS) system-a Cloud-hosted metadata retrieval, caching, and prefetching system, and have evaluated its performance on Amazon EC2 and XSEDE.

KW - Caching

KW - Metadata retrieval

KW - Prefetching

KW - Software as a service (SaaS)

KW - Wide-area transfers

UR - http://www.scopus.com/inward/record.url?scp=84892932155&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84892932155&partnerID=8YFLogxK

U2 - 10.1145/2534695.2534700

DO - 10.1145/2534695.2534700

M3 - Conference contribution

AN - SCOPUS:84892932155

SN - 9781450325226

T3 - Proc. of NDM 2013: 3rd Int. Workshop on Network-Aware Data Management - Held in Conjunction with SC 2013: The Int. Conference for High Performance Computing, Networking, Storage and Analysis

BT - Proc. of NDM 2013

ER -