Predictive data and energy management in GreenHDFS

Rini T. Kaushik, Tarek Abdelzaher, Ryota Egashira, Klara Nahrstedt

Research output: Chapter in Book/Report/Conference proceedingConference contribution


The sheer scale and rapid rise of Big Data mandates highly scalable, self-adaptive, and energy-conserving data-intensive compute clusters. Based on our analysis of the traces from a production Hadoop cluster at Yahoo!, we observe that file size, file lifespan, and file heat are statistically correlated and very strongly associated with the hierarchical directory structure (i.e., absolute file path) in which the files are organized. Leveraging that observation, we present predictive GreenHDFS; an energy-conserving variant of the Hadoop distributed file system that uses a supervised machine learning technique to learn the correlation between the directory hierarchy and the file attributes to guide novel predictive file zone placement, migration, and replication policies that significantly outperform the current state-of-the-art reactive approaches. Using real-world traces from a large-scale (2600 servers, 5 Petabytes) production Hadoop cluster at Yahoo! in our GreenHDFS simulations, we show how predictive GreenHDFS results in a much better trade-off between performance and energy consumption.

Original languageEnglish (US)
Title of host publication2011 International Green Computing Conference and Workshops, IGCC 2011
StatePublished - 2011
Event2011 International Green Computing Conference, IGCC 2011 - Orlando, FL, United States
Duration: Jul 25 2011Jul 28 2011

Publication series

Name2011 International Green Computing Conference and Workshops, IGCC 2011


Other2011 International Green Computing Conference, IGCC 2011
Country/TerritoryUnited States
CityOrlando, FL

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Environmental Engineering


Dive into the research topics of 'Predictive data and energy management in GreenHDFS'. Together they form a unique fingerprint.

Cite this