Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters

Brian Cho, Muntasir Rahman, Tej Chajed, Indranil Gupta, Cristina Abad, Nathan Roberts, Philbert Lin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) exploration and evaluation of smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism for Mapreduce. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! Inc. commercial cluster, using both synthetic workloads as well as Hadoop cluster traces from Yahoo!. Our results reveal that Natjam incurs overheads as low as 7%, and is preferable to existing approaches.

Original languageEnglish (US)
Title of host publicationProceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013
PublisherAssociation for Computing Machinery
ISBN (Print)9781450324281
DOIs
StatePublished - Jan 1 2013
Event4th Annual Symposium on Cloud Computing, SoCC 2013 - Santa Clara, CA, United States
Duration: Oct 1 2013Oct 3 2013

Publication series

NameProceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013

Other

Other4th Annual Symposium on Cloud Computing, SoCC 2013
CountryUnited States
CitySanta Clara, CA
Period10/1/1310/3/13

Fingerprint

Scheduling
Experiments

Keywords

  • Deadlines
  • Hadoop
  • Mapreduce
  • Priorities
  • Scheduling

ASJC Scopus subject areas

  • Software

Cite this

Cho, B., Rahman, M., Chajed, T., Gupta, I., Abad, C., Roberts, N., & Lin, P. (2013). Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. In Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013 [6] (Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013). Association for Computing Machinery. https://doi.org/10.1145/2523616.2523624

Natjam : Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. / Cho, Brian; Rahman, Muntasir; Chajed, Tej; Gupta, Indranil; Abad, Cristina; Roberts, Nathan; Lin, Philbert.

Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013. Association for Computing Machinery, 2013. 6 (Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cho, B, Rahman, M, Chajed, T, Gupta, I, Abad, C, Roberts, N & Lin, P 2013, Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. in Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013., 6, Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013, Association for Computing Machinery, 4th Annual Symposium on Cloud Computing, SoCC 2013, Santa Clara, CA, United States, 10/1/13. https://doi.org/10.1145/2523616.2523624
Cho B, Rahman M, Chajed T, Gupta I, Abad C, Roberts N et al. Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. In Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013. Association for Computing Machinery. 2013. 6. (Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013). https://doi.org/10.1145/2523616.2523624
Cho, Brian ; Rahman, Muntasir ; Chajed, Tej ; Gupta, Indranil ; Abad, Cristina ; Roberts, Nathan ; Lin, Philbert. / Natjam : Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013. Association for Computing Machinery, 2013. (Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013).
@inproceedings{89e26cdafceb4fa3add05af987648a2d,
title = "Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters",
abstract = "This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) exploration and evaluation of smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism for Mapreduce. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! Inc. commercial cluster, using both synthetic workloads as well as Hadoop cluster traces from Yahoo!. Our results reveal that Natjam incurs overheads as low as 7{\%}, and is preferable to existing approaches.",
keywords = "Deadlines, Hadoop, Mapreduce, Priorities, Scheduling",
author = "Brian Cho and Muntasir Rahman and Tej Chajed and Indranil Gupta and Cristina Abad and Nathan Roberts and Philbert Lin",
year = "2013",
month = "1",
day = "1",
doi = "10.1145/2523616.2523624",
language = "English (US)",
isbn = "9781450324281",
series = "Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013",
publisher = "Association for Computing Machinery",
booktitle = "Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013",

}

TY - GEN

T1 - Natjam

T2 - Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters

AU - Cho, Brian

AU - Rahman, Muntasir

AU - Chajed, Tej

AU - Gupta, Indranil

AU - Abad, Cristina

AU - Roberts, Nathan

AU - Lin, Philbert

PY - 2013/1/1

Y1 - 2013/1/1

N2 - This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) exploration and evaluation of smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism for Mapreduce. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! Inc. commercial cluster, using both synthetic workloads as well as Hadoop cluster traces from Yahoo!. Our results reveal that Natjam incurs overheads as low as 7%, and is preferable to existing approaches.

AB - This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) exploration and evaluation of smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism for Mapreduce. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! Inc. commercial cluster, using both synthetic workloads as well as Hadoop cluster traces from Yahoo!. Our results reveal that Natjam incurs overheads as low as 7%, and is preferable to existing approaches.

KW - Deadlines

KW - Hadoop

KW - Mapreduce

KW - Priorities

KW - Scheduling

UR - http://www.scopus.com/inward/record.url?scp=84893335405&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84893335405&partnerID=8YFLogxK

U2 - 10.1145/2523616.2523624

DO - 10.1145/2523616.2523624

M3 - Conference contribution

AN - SCOPUS:84893335405

SN - 9781450324281

T3 - Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013

BT - Proceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013

PB - Association for Computing Machinery

ER -