OPTiC: Opportunistic graph processing in multi-tenant clusters

Muntasir Raihan Rahman, Indranil Gupta, Akash Kapoor, Haozhen Ding

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present OPTiC, a multi-tenant scheduler intended for distributed graph processing frameworks. OPTiC proposes opportunistic scheduling, whereby queued jobs can be pre-scheduled at cluster nodes when the cluster is fully busy running jobs. This allows overlapping of data ingress with ongoing computation. To pre-schedule wisely, OPTiC's novel contribution is a profile-free and cluster-agnostic approach to compare progress of graph processing jobs. OPTiC is implemented inside Apache Giraph, with YARN underneath. Our experiments with real workload traces and network models show that OPTiC's opportunistic scheduling improves run time (both at the median and at the tail) by 20%-82% compared to baseline multi-tenancy, in a variety of scenarios.

Original languageEnglish (US)
Title of host publicationProceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018
EditorsJie Li, Abhishek Chandra, Tian Guo, Ying Cai
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages113-123
Number of pages11
ISBN (Electronic)9781538650080
DOIs
StatePublished - May 16 2018
Event2018 IEEE International Conference on Cloud Engineering, IC2E 2018 - Orlando, United States
Duration: Apr 17 2018Apr 20 2018

Publication series

NameProceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018

Other

Other2018 IEEE International Conference on Cloud Engineering, IC2E 2018
CountryUnited States
CityOrlando
Period4/17/184/20/18

Keywords

  • Cluster scheduling
  • Graph-processing
  • Multi-tenancy

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'OPTiC: Opportunistic graph processing in multi-tenant clusters'. Together they form a unique fingerprint.

  • Cite this

    Rahman, M. R., Gupta, I., Kapoor, A., & Ding, H. (2018). OPTiC: Opportunistic graph processing in multi-tenant clusters. In J. Li, A. Chandra, T. Guo, & Y. Cai (Eds.), Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018 (pp. 113-123). (Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IC2E.2018.00034