Henge

Intent-driven multi-tenant stream processing

Faria Kalim, Le Xu, Sharanya Bathey, Richa Meherwal, Indranil Gupta

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present Henge, a system to support intent-based multi-tenancy in modern distributed stream processing systems. Henge supports multi-tenancy as a first-class citizen: everyone in an organization can now submit their stream processing jobs to a single, shared, consolidated cluster. Secondly, Henge allows each job to specify its own intents (i.e., requirements) as a Service Level Objective (SLO) that captures latency and/or throughput needs. In such an intent-driven multi-tenant cluster, the Henge scheduler adapts continually to meet jobs’ respective SLOs in spite of limited cluster resources, and under dynamically varying workloads. SLOs are soft and are based on utility functions. Henge’s overall goal is to maximize the total system utility achieved by all jobs in the system. Henge is integrated into Apache Storm and we present experimental results using both production jobs from Yahoo! and real datasets from Twitter.

Original languageEnglish (US)
Title of host publicationSoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing
PublisherAssociation for Computing Machinery, Inc
Pages249-262
Number of pages14
ISBN (Electronic)9781450360111
DOIs
StatePublished - Oct 11 2018
Event2018 ACM Symposium on Cloud Computing, SoCC 2018 - Carlsbad, United States
Duration: Oct 11 2018Oct 13 2018

Publication series

NameSoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing

Other

Other2018 ACM Symposium on Cloud Computing, SoCC 2018
CountryUnited States
CityCarlsbad
Period10/11/1810/13/18

Fingerprint

Processing
Throughput

Keywords

  • Intents
  • Multi-Tenancy
  • Resource Management
  • Service Level Objectives
  • Stream Processing

ASJC Scopus subject areas

  • Artificial Intelligence
  • Information Systems
  • Software

Cite this

Kalim, F., Xu, L., Bathey, S., Meherwal, R., & Gupta, I. (2018). Henge: Intent-driven multi-tenant stream processing. In SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing (pp. 249-262). (SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing). Association for Computing Machinery, Inc. https://doi.org/10.1145/3267809.3267832

Henge : Intent-driven multi-tenant stream processing. / Kalim, Faria; Xu, Le; Bathey, Sharanya; Meherwal, Richa; Gupta, Indranil.

SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing. Association for Computing Machinery, Inc, 2018. p. 249-262 (SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kalim, F, Xu, L, Bathey, S, Meherwal, R & Gupta, I 2018, Henge: Intent-driven multi-tenant stream processing. in SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing. SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing, Association for Computing Machinery, Inc, pp. 249-262, 2018 ACM Symposium on Cloud Computing, SoCC 2018, Carlsbad, United States, 10/11/18. https://doi.org/10.1145/3267809.3267832
Kalim F, Xu L, Bathey S, Meherwal R, Gupta I. Henge: Intent-driven multi-tenant stream processing. In SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing. Association for Computing Machinery, Inc. 2018. p. 249-262. (SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing). https://doi.org/10.1145/3267809.3267832
Kalim, Faria ; Xu, Le ; Bathey, Sharanya ; Meherwal, Richa ; Gupta, Indranil. / Henge : Intent-driven multi-tenant stream processing. SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing. Association for Computing Machinery, Inc, 2018. pp. 249-262 (SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing).
@inproceedings{8a3594f715af4c08983ba3a49b2f6a66,
title = "Henge: Intent-driven multi-tenant stream processing",
abstract = "We present Henge, a system to support intent-based multi-tenancy in modern distributed stream processing systems. Henge supports multi-tenancy as a first-class citizen: everyone in an organization can now submit their stream processing jobs to a single, shared, consolidated cluster. Secondly, Henge allows each job to specify its own intents (i.e., requirements) as a Service Level Objective (SLO) that captures latency and/or throughput needs. In such an intent-driven multi-tenant cluster, the Henge scheduler adapts continually to meet jobs’ respective SLOs in spite of limited cluster resources, and under dynamically varying workloads. SLOs are soft and are based on utility functions. Henge’s overall goal is to maximize the total system utility achieved by all jobs in the system. Henge is integrated into Apache Storm and we present experimental results using both production jobs from Yahoo! and real datasets from Twitter.",
keywords = "Intents, Multi-Tenancy, Resource Management, Service Level Objectives, Stream Processing",
author = "Faria Kalim and Le Xu and Sharanya Bathey and Richa Meherwal and Indranil Gupta",
year = "2018",
month = "10",
day = "11",
doi = "10.1145/3267809.3267832",
language = "English (US)",
series = "SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing",
publisher = "Association for Computing Machinery, Inc",
pages = "249--262",
booktitle = "SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing",

}

TY - GEN

T1 - Henge

T2 - Intent-driven multi-tenant stream processing

AU - Kalim, Faria

AU - Xu, Le

AU - Bathey, Sharanya

AU - Meherwal, Richa

AU - Gupta, Indranil

PY - 2018/10/11

Y1 - 2018/10/11

N2 - We present Henge, a system to support intent-based multi-tenancy in modern distributed stream processing systems. Henge supports multi-tenancy as a first-class citizen: everyone in an organization can now submit their stream processing jobs to a single, shared, consolidated cluster. Secondly, Henge allows each job to specify its own intents (i.e., requirements) as a Service Level Objective (SLO) that captures latency and/or throughput needs. In such an intent-driven multi-tenant cluster, the Henge scheduler adapts continually to meet jobs’ respective SLOs in spite of limited cluster resources, and under dynamically varying workloads. SLOs are soft and are based on utility functions. Henge’s overall goal is to maximize the total system utility achieved by all jobs in the system. Henge is integrated into Apache Storm and we present experimental results using both production jobs from Yahoo! and real datasets from Twitter.

AB - We present Henge, a system to support intent-based multi-tenancy in modern distributed stream processing systems. Henge supports multi-tenancy as a first-class citizen: everyone in an organization can now submit their stream processing jobs to a single, shared, consolidated cluster. Secondly, Henge allows each job to specify its own intents (i.e., requirements) as a Service Level Objective (SLO) that captures latency and/or throughput needs. In such an intent-driven multi-tenant cluster, the Henge scheduler adapts continually to meet jobs’ respective SLOs in spite of limited cluster resources, and under dynamically varying workloads. SLOs are soft and are based on utility functions. Henge’s overall goal is to maximize the total system utility achieved by all jobs in the system. Henge is integrated into Apache Storm and we present experimental results using both production jobs from Yahoo! and real datasets from Twitter.

KW - Intents

KW - Multi-Tenancy

KW - Resource Management

KW - Service Level Objectives

KW - Stream Processing

UR - http://www.scopus.com/inward/record.url?scp=85059004880&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85059004880&partnerID=8YFLogxK

U2 - 10.1145/3267809.3267832

DO - 10.1145/3267809.3267832

M3 - Conference contribution

T3 - SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing

SP - 249

EP - 262

BT - SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing

PB - Association for Computing Machinery, Inc

ER -