A storage-centric analysis of MapReduce workloads: File popularity, temporal locality and arrival patterns

Cristina L. Abad, Nathan Roberts, Yi Lu, Roy H. Campbell

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

A huge increase in data storage and processing requirements has lead to Big Data, for which next generation storage systems are being designed and implemented. However, we have a limited understanding of the workloads of Big Data storage systems. We consider the case of one common type of Big Data storage cluster: a cluster dedicated to supporting a mix of MapReduce jobs. We analyze 6-month traces from two large Hadoop clusters at Yahoo! and characterize the file popularity, temporal locality, and arrival patterns of the workloads. We identify several interesting properties and compare them with previous observations from web and media server workloads. To the best of our knowledge, this is the first study of how MapReduce workloads interact with the storage layer.

Original languageEnglish (US)
Title of host publicationProceedings - 2012 IEEE International Symposium on Workload Characterization, IISWC 2012
PublisherIEEE Computer Society
Pages100-109
Number of pages10
ISBN (Print)9781457720642
DOIs
StatePublished - 2012
Event2012 IEEE International Symposium on Workload Characterization, IISWC 2012 - San Diego, CA, United States
Duration: Nov 4 2012Nov 6 2012

Publication series

NameProceedings - 2012 IEEE International Symposium on Workload Characterization, IISWC 2012

Other

Other2012 IEEE International Symposium on Workload Characterization, IISWC 2012
Country/TerritoryUnited States
CitySan Diego, CA
Period11/4/1211/6/12

Keywords

  • Access patterns
  • Big Data
  • HDFS
  • MapReduce

ASJC Scopus subject areas

  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'A storage-centric analysis of MapReduce workloads: File popularity, temporal locality and arrival patterns'. Together they form a unique fingerprint.

Cite this