Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters

Brian Cho, Muntasir Rahman, Tej Chajed, Indranil Gupta, Cristina Abad, Nathan Roberts, Philbert Lin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) exploration and evaluation of smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism for Mapreduce. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! Inc. commercial cluster, using both synthetic workloads as well as Hadoop cluster traces from Yahoo!. Our results reveal that Natjam incurs overheads as low as 7%, and is preferable to existing approaches.

Original languageEnglish (US)
Title of host publicationProceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013
PublisherAssociation for Computing Machinery
ISBN (Print)9781450324281
DOIs
StatePublished - Jan 1 2013
Event4th Annual Symposium on Cloud Computing, SoCC 2013 - Santa Clara, CA, United States
Duration: Oct 1 2013Oct 3 2013

Publication series

NameProceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013

Other

Other4th Annual Symposium on Cloud Computing, SoCC 2013
CountryUnited States
CitySanta Clara, CA
Period10/1/1310/3/13

    Fingerprint

Keywords

  • Deadlines
  • Hadoop
  • Mapreduce
  • Priorities
  • Scheduling

ASJC Scopus subject areas

  • Software

Cite this

Cho, B., Rahman, M., Chajed, T., Gupta, I., Abad, C., Roberts, N., & Lin, P. (2013). Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. In Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013 [6] (Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013). Association for Computing Machinery. https://doi.org/10.1145/2523616.2523624