Play it again, SimMR!

A typical Map Reduce cluster is shared among different users and multiple applications. A challenging problem in such shared environments is the ability to efficiently control resource allocations among the running and submitted jobs for achieving users' performance goals. To ease the task of evaluating and comparing different provisioning and scheduling approaches in MapReduce environments, we have designed and implemented a simulation environment SimMR ¹ which is comprised of three inter-related components: i) Trace Generator that creates a replayable Map Reduce workload; ii) Simulator Engine that accurately emulates the job master functionality in Hadoop; and iii) a pluggable scheduling policy that dictates the scheduler decisions on job ordering and the amount of resources allocated to different jobs over time. We validate the accuracy of SimMR environment by, first, executing a set of realistic MapReduce applications in a 66-node Hadoop cluster and then by replaying the collected job execution traces in SimMR. Our simulator accurately reproduces the original job processing: the completion times of the simulated jobs are within 5% of the original ones. SimMR can process over one million events per second. This allows users to simulate complex workloads in a few seconds instead of multi-hour executions in the real testbed. Finally, by using SimMR we analyze and compare the performance of two novel deadline-driven schedulers over a diverse set of real and synthetic workloads.