Phurti: Application and network-aware flow scheduling for multi-tenant MapReduce clusters

Chris X. Cai, Shayan Saeed, Indranil Gupta, Roy H. Campbell, Franck Le

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Traffic for a typical MapReduce job in a data center consists of multiple network flows. Traditionally, network resources have been allocated to optimize network-level metrics such as flow completion time or throughput. Some recent schemes propose using application-aware scheduling which can shorten the average job completion time. However, most of them treat the core network as a black box with sufficient capacity. Even if only one network link in the core network becomes a bottleneck, it can hurt application performance. We design and implement a centralized flow-scheduling framework called Phurti with the goal of improving the completion time for jobs in a cluster shared among multiple Hadoop jobs (multi-tenant). Phurti communicates both with the Hadoop framework to retrieve job-level network traffic information and the OpenFlow-based switches to learn about the network topology. Phurti implements a novel heuristic called Smallest Maximum Sequential-traffic First (SMSF) that uses collected application and network information to perform traffic scheduling for MapReduce jobs. Our evaluation with real Hadoop workloads shows that compared to application and network-agnostic scheduling strategies, Phurti improves job completion time for 95% of the jobs, decreases average job completion time by 20%, tail job completion time by 13% and scales well with the cluster size and number of jobs.

Original languageEnglish (US)
Title of host publicationProceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016
Subtitle of host publicationCo-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages161-170
Number of pages10
ISBN (Electronic)9781509019618
DOIs
StatePublished - Jun 1 2016
Event4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016 - Berlin, Germany
Duration: Apr 4 2016Apr 8 2016

Publication series

NameProceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016

Other

Other4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016
CountryGermany
CityBerlin
Period4/4/164/8/16

Keywords

  • MapReduce
  • SDN
  • network
  • scheduling

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Phurti: Application and network-aware flow scheduling for multi-tenant MapReduce clusters'. Together they form a unique fingerprint.

  • Cite this

    Cai, C. X., Saeed, S., Gupta, I., Campbell, R. H., & Le, F. (2016). Phurti: Application and network-aware flow scheduling for multi-tenant MapReduce clusters. In Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016 (pp. 161-170). [7484180] (Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IC2E.2016.21