OPTiC: Opportunistic graph processing in multi-tenant clusters

Muntasir Raihan Rahman, Indranil Gupta, Akash Kapoor, Haozhen Ding

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present OPTiC, a multi-tenant scheduler intended for distributed graph processing frameworks. OPTiC proposes opportunistic scheduling, whereby queued jobs can be pre-scheduled at cluster nodes when the cluster is fully busy running jobs. This allows overlapping of data ingress with ongoing computation. To pre-schedule wisely, OPTiC's novel contribution is a profile-free and cluster-agnostic approach to compare progress of graph processing jobs. OPTiC is implemented inside Apache Giraph, with YARN underneath. Our experiments with real workload traces and network models show that OPTiC's opportunistic scheduling improves run time (both at the median and at the tail) by 20%-82% compared to baseline multi-tenancy, in a variety of scenarios.

Original languageEnglish (US)
Title of host publicationProceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages113-123
Number of pages11
ISBN (Electronic)9781538650080
DOIs
StatePublished - May 16 2018
Event2018 IEEE International Conference on Cloud Engineering, IC2E 2018 - Orlando, United States
Duration: Apr 17 2018Apr 20 2018

Other

Other2018 IEEE International Conference on Cloud Engineering, IC2E 2018
CountryUnited States
CityOrlando
Period4/17/184/20/18

Fingerprint

Scheduling
Processing
Experiments

Keywords

  • Cluster scheduling
  • Graph-processing
  • Multi-tenancy

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture

Cite this

Rahman, M. R., Gupta, I., Kapoor, A., & Ding, H. (2018). OPTiC: Opportunistic graph processing in multi-tenant clusters. In Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018 (pp. 113-123). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IC2E.2018.00034

OPTiC : Opportunistic graph processing in multi-tenant clusters. / Rahman, Muntasir Raihan; Gupta, Indranil; Kapoor, Akash; Ding, Haozhen.

Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 113-123.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Rahman, MR, Gupta, I, Kapoor, A & Ding, H 2018, OPTiC: Opportunistic graph processing in multi-tenant clusters. in Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018. Institute of Electrical and Electronics Engineers Inc., pp. 113-123, 2018 IEEE International Conference on Cloud Engineering, IC2E 2018, Orlando, United States, 4/17/18. https://doi.org/10.1109/IC2E.2018.00034
Rahman MR, Gupta I, Kapoor A, Ding H. OPTiC: Opportunistic graph processing in multi-tenant clusters. In Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 113-123 https://doi.org/10.1109/IC2E.2018.00034
Rahman, Muntasir Raihan ; Gupta, Indranil ; Kapoor, Akash ; Ding, Haozhen. / OPTiC : Opportunistic graph processing in multi-tenant clusters. Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 113-123
@inproceedings{a444d98b061e411682c4b6da6dbb5688,
title = "OPTiC: Opportunistic graph processing in multi-tenant clusters",
abstract = "We present OPTiC, a multi-tenant scheduler intended for distributed graph processing frameworks. OPTiC proposes opportunistic scheduling, whereby queued jobs can be pre-scheduled at cluster nodes when the cluster is fully busy running jobs. This allows overlapping of data ingress with ongoing computation. To pre-schedule wisely, OPTiC's novel contribution is a profile-free and cluster-agnostic approach to compare progress of graph processing jobs. OPTiC is implemented inside Apache Giraph, with YARN underneath. Our experiments with real workload traces and network models show that OPTiC's opportunistic scheduling improves run time (both at the median and at the tail) by 20{\%}-82{\%} compared to baseline multi-tenancy, in a variety of scenarios.",
keywords = "Cluster scheduling, Graph-processing, Multi-tenancy",
author = "Rahman, {Muntasir Raihan} and Indranil Gupta and Akash Kapoor and Haozhen Ding",
year = "2018",
month = "5",
day = "16",
doi = "10.1109/IC2E.2018.00034",
language = "English (US)",
pages = "113--123",
booktitle = "Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
address = "United States",

}

TY - GEN

T1 - OPTiC

T2 - Opportunistic graph processing in multi-tenant clusters

AU - Rahman, Muntasir Raihan

AU - Gupta, Indranil

AU - Kapoor, Akash

AU - Ding, Haozhen

PY - 2018/5/16

Y1 - 2018/5/16

N2 - We present OPTiC, a multi-tenant scheduler intended for distributed graph processing frameworks. OPTiC proposes opportunistic scheduling, whereby queued jobs can be pre-scheduled at cluster nodes when the cluster is fully busy running jobs. This allows overlapping of data ingress with ongoing computation. To pre-schedule wisely, OPTiC's novel contribution is a profile-free and cluster-agnostic approach to compare progress of graph processing jobs. OPTiC is implemented inside Apache Giraph, with YARN underneath. Our experiments with real workload traces and network models show that OPTiC's opportunistic scheduling improves run time (both at the median and at the tail) by 20%-82% compared to baseline multi-tenancy, in a variety of scenarios.

AB - We present OPTiC, a multi-tenant scheduler intended for distributed graph processing frameworks. OPTiC proposes opportunistic scheduling, whereby queued jobs can be pre-scheduled at cluster nodes when the cluster is fully busy running jobs. This allows overlapping of data ingress with ongoing computation. To pre-schedule wisely, OPTiC's novel contribution is a profile-free and cluster-agnostic approach to compare progress of graph processing jobs. OPTiC is implemented inside Apache Giraph, with YARN underneath. Our experiments with real workload traces and network models show that OPTiC's opportunistic scheduling improves run time (both at the median and at the tail) by 20%-82% compared to baseline multi-tenancy, in a variety of scenarios.

KW - Cluster scheduling

KW - Graph-processing

KW - Multi-tenancy

UR - http://www.scopus.com/inward/record.url?scp=85048348885&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048348885&partnerID=8YFLogxK

U2 - 10.1109/IC2E.2018.00034

DO - 10.1109/IC2E.2018.00034

M3 - Conference contribution

AN - SCOPUS:85048348885

SP - 113

EP - 123

BT - Proceedings - 2018 IEEE International Conference on Cloud Engineering, IC2E 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -