Play it again, SimMR!

Abhishek Verma, Ludmila Cherkasova, R H Campbell

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

A typical MapReduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we designed and implemented a simulation environment Sim MR which is comprised of three inter-related components: i) Trace Generator that creates a replayable MapReduce workload, ii) Simulator Engine that accurately emulates the job master functionality in Hadoop, and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of Sim MR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5% of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real test bed. Finally, by using SimMR we analyze and compare performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.

Original languageEnglish (US)
Title of host publicationProceedings - 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011
Pages253-261
Number of pages9
DOIs
StatePublished - Nov 16 2011
Event2011 IEEE International Conference on Cluster Computing, CLUSTER 2011 - Austin, TX, United States
Duration: Sep 26 2011Sep 30 2011

Publication series

NameProceedings - IEEE International Conference on Cluster Computing, ICCC
ISSN (Print)1552-5244

Other

Other2011 IEEE International Conference on Cluster Computing, CLUSTER 2011
CountryUnited States
CityAustin, TX
Period9/26/119/30/11

Fingerprint

Simulators
Scheduling
Resource allocation
Engines
Processing

Keywords

  • MapReduce
  • Schedulers
  • Simulator
  • Traces

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Signal Processing

Cite this

Verma, A., Cherkasova, L., & Campbell, R. H. (2011). Play it again, SimMR! In Proceedings - 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011 (pp. 253-261). [6061143] (Proceedings - IEEE International Conference on Cluster Computing, ICCC). https://doi.org/10.1109/CLUSTER.2011.36

Play it again, SimMR! / Verma, Abhishek; Cherkasova, Ludmila; Campbell, R H.

Proceedings - 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011. 2011. p. 253-261 6061143 (Proceedings - IEEE International Conference on Cluster Computing, ICCC).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Verma, A, Cherkasova, L & Campbell, RH 2011, Play it again, SimMR! in Proceedings - 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011., 6061143, Proceedings - IEEE International Conference on Cluster Computing, ICCC, pp. 253-261, 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011, Austin, TX, United States, 9/26/11. https://doi.org/10.1109/CLUSTER.2011.36
Verma A, Cherkasova L, Campbell RH. Play it again, SimMR! In Proceedings - 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011. 2011. p. 253-261. 6061143. (Proceedings - IEEE International Conference on Cluster Computing, ICCC). https://doi.org/10.1109/CLUSTER.2011.36
Verma, Abhishek ; Cherkasova, Ludmila ; Campbell, R H. / Play it again, SimMR!. Proceedings - 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011. 2011. pp. 253-261 (Proceedings - IEEE International Conference on Cluster Computing, ICCC).
@inproceedings{8a1c482097e045c3a035dce98064230b,
title = "Play it again, SimMR!",
abstract = "A typical MapReduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we designed and implemented a simulation environment Sim MR which is comprised of three inter-related components: i) Trace Generator that creates a replayable MapReduce workload, ii) Simulator Engine that accurately emulates the job master functionality in Hadoop, and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of Sim MR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5{\%} of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real test bed. Finally, by using SimMR we analyze and compare performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.",
keywords = "MapReduce, Schedulers, Simulator, Traces",
author = "Abhishek Verma and Ludmila Cherkasova and Campbell, {R H}",
year = "2011",
month = "11",
day = "16",
doi = "10.1109/CLUSTER.2011.36",
language = "English (US)",
isbn = "9780769545165",
series = "Proceedings - IEEE International Conference on Cluster Computing, ICCC",
pages = "253--261",
booktitle = "Proceedings - 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011",

}

TY - GEN

T1 - Play it again, SimMR!

AU - Verma, Abhishek

AU - Cherkasova, Ludmila

AU - Campbell, R H

PY - 2011/11/16

Y1 - 2011/11/16

N2 - A typical MapReduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we designed and implemented a simulation environment Sim MR which is comprised of three inter-related components: i) Trace Generator that creates a replayable MapReduce workload, ii) Simulator Engine that accurately emulates the job master functionality in Hadoop, and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of Sim MR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5% of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real test bed. Finally, by using SimMR we analyze and compare performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.

AB - A typical MapReduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we designed and implemented a simulation environment Sim MR which is comprised of three inter-related components: i) Trace Generator that creates a replayable MapReduce workload, ii) Simulator Engine that accurately emulates the job master functionality in Hadoop, and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of Sim MR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5% of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real test bed. Finally, by using SimMR we analyze and compare performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.

KW - MapReduce

KW - Schedulers

KW - Simulator

KW - Traces

UR - http://www.scopus.com/inward/record.url?scp=80955167912&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80955167912&partnerID=8YFLogxK

U2 - 10.1109/CLUSTER.2011.36

DO - 10.1109/CLUSTER.2011.36

M3 - Conference contribution

AN - SCOPUS:80955167912

SN - 9780769545165

T3 - Proceedings - IEEE International Conference on Cluster Computing, ICCC

SP - 253

EP - 261

BT - Proceedings - 2011 IEEE International Conference on Cluster Computing, CLUSTER 2011

ER -