Play it again, SimMR!

Abhishek Verma, Ludmila Cherkasova, R H Campbell

Research output: Contribution to journalArticle

Abstract

A typical Map Reduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we have designed and implemented a simulation environment SimMR 1 which is comprised of three inter-related components: i) Trace Generator that creates a replayable Map Reduce workload; ii) Simulator Engine that accurately emulates the job master functionality in Hadoop; and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of SimMR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5% of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real testbed. Finally, by using SimMR we analyze and compare the performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.

Original languageEnglish (US)
JournalHP Laboratories Technical Report
Issue number127
StatePublished - Aug 31 2011

Fingerprint

Simulators
Scheduling
Testbeds
Resource allocation
Engines
Processing

Keywords

  • Map reduce
  • Schedulers
  • Simulator
  • Traces

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Cite this

Play it again, SimMR! / Verma, Abhishek; Cherkasova, Ludmila; Campbell, R H.

In: HP Laboratories Technical Report, No. 127, 31.08.2011.

Research output: Contribution to journalArticle

Verma A, Cherkasova L, Campbell RH. Play it again, SimMR! HP Laboratories Technical Report. 2011 Aug 31;(127).
Verma, Abhishek ; Cherkasova, Ludmila ; Campbell, R H. / Play it again, SimMR!. In: HP Laboratories Technical Report. 2011 ; No. 127.
@article{42fe9f26590f4472809da4ab2f23999d,
title = "Play it again, SimMR!",
abstract = "A typical Map Reduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we have designed and implemented a simulation environment SimMR 1 which is comprised of three inter-related components: i) Trace Generator that creates a replayable Map Reduce workload; ii) Simulator Engine that accurately emulates the job master functionality in Hadoop; and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of SimMR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5{\%} of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real testbed. Finally, by using SimMR we analyze and compare the performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.",
keywords = "Map reduce, Schedulers, Simulator, Traces",
author = "Abhishek Verma and Ludmila Cherkasova and Campbell, {R H}",
year = "2011",
month = "8",
day = "31",
language = "English (US)",
journal = "HP Laboratories Technical Report",
number = "127",

}

TY - JOUR

T1 - Play it again, SimMR!

AU - Verma, Abhishek

AU - Cherkasova, Ludmila

AU - Campbell, R H

PY - 2011/8/31

Y1 - 2011/8/31

N2 - A typical Map Reduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we have designed and implemented a simulation environment SimMR 1 which is comprised of three inter-related components: i) Trace Generator that creates a replayable Map Reduce workload; ii) Simulator Engine that accurately emulates the job master functionality in Hadoop; and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of SimMR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5% of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real testbed. Finally, by using SimMR we analyze and compare the performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.

AB - A typical Map Reduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we have designed and implemented a simulation environment SimMR 1 which is comprised of three inter-related components: i) Trace Generator that creates a replayable Map Reduce workload; ii) Simulator Engine that accurately emulates the job master functionality in Hadoop; and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of SimMR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5% of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real testbed. Finally, by using SimMR we analyze and compare the performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.

KW - Map reduce

KW - Schedulers

KW - Simulator

KW - Traces

UR - http://www.scopus.com/inward/record.url?scp=80052087041&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80052087041&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:80052087041

JO - HP Laboratories Technical Report

JF - HP Laboratories Technical Report

IS - 127

ER -