Abstract
We present Henge, a system to support intent-based multi-tenancy in modern distributed stream processing systems. Henge supports multi-tenancy as a first-class citizen: everyone in an organization can now submit their stream processing jobs to a single, shared, consolidated cluster. Secondly, Henge allows each job to specify its own intents (i.e., requirements) as a Service Level Objective (SLO) that captures latency and/or throughput needs. In such an intent-driven multi-tenant cluster, the Henge scheduler adapts continually to meet jobs’ respective SLOs in spite of limited cluster resources, and under dynamically varying workloads. SLOs are soft and are based on utility functions. Henge’s overall goal is to maximize the total system utility achieved by all jobs in the system. Henge is integrated into Apache Storm and we present experimental results using both production jobs from Yahoo! and real datasets from Twitter.
| Original language | English (US) |
|---|---|
| Title of host publication | SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing |
| Publisher | Association for Computing Machinery, Inc |
| Pages | 249-262 |
| Number of pages | 14 |
| ISBN (Electronic) | 9781450360111 |
| DOIs | |
| State | Published - Oct 11 2018 |
| Event | 2018 ACM Symposium on Cloud Computing, SoCC 2018 - Carlsbad, United States Duration: Oct 11 2018 → Oct 13 2018 |
Publication series
| Name | SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing |
|---|
Other
| Other | 2018 ACM Symposium on Cloud Computing, SoCC 2018 |
|---|---|
| Country | United States |
| City | Carlsbad |
| Period | 10/11/18 → 10/13/18 |
Fingerprint
Keywords
- Intents
- Multi-Tenancy
- Resource Management
- Service Level Objectives
- Stream Processing
ASJC Scopus subject areas
- Artificial Intelligence
- Information Systems
- Software
Cite this
Henge : Intent-driven multi-tenant stream processing. / Kalim, Faria; Xu, Le; Bathey, Sharanya; Meherwal, Richa; Gupta, Indranil.
SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing. Association for Computing Machinery, Inc, 2018. p. 249-262 (SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Henge
T2 - Intent-driven multi-tenant stream processing
AU - Kalim, Faria
AU - Xu, Le
AU - Bathey, Sharanya
AU - Meherwal, Richa
AU - Gupta, Indranil
PY - 2018/10/11
Y1 - 2018/10/11
N2 - We present Henge, a system to support intent-based multi-tenancy in modern distributed stream processing systems. Henge supports multi-tenancy as a first-class citizen: everyone in an organization can now submit their stream processing jobs to a single, shared, consolidated cluster. Secondly, Henge allows each job to specify its own intents (i.e., requirements) as a Service Level Objective (SLO) that captures latency and/or throughput needs. In such an intent-driven multi-tenant cluster, the Henge scheduler adapts continually to meet jobs’ respective SLOs in spite of limited cluster resources, and under dynamically varying workloads. SLOs are soft and are based on utility functions. Henge’s overall goal is to maximize the total system utility achieved by all jobs in the system. Henge is integrated into Apache Storm and we present experimental results using both production jobs from Yahoo! and real datasets from Twitter.
AB - We present Henge, a system to support intent-based multi-tenancy in modern distributed stream processing systems. Henge supports multi-tenancy as a first-class citizen: everyone in an organization can now submit their stream processing jobs to a single, shared, consolidated cluster. Secondly, Henge allows each job to specify its own intents (i.e., requirements) as a Service Level Objective (SLO) that captures latency and/or throughput needs. In such an intent-driven multi-tenant cluster, the Henge scheduler adapts continually to meet jobs’ respective SLOs in spite of limited cluster resources, and under dynamically varying workloads. SLOs are soft and are based on utility functions. Henge’s overall goal is to maximize the total system utility achieved by all jobs in the system. Henge is integrated into Apache Storm and we present experimental results using both production jobs from Yahoo! and real datasets from Twitter.
KW - Intents
KW - Multi-Tenancy
KW - Resource Management
KW - Service Level Objectives
KW - Stream Processing
UR - http://www.scopus.com/inward/record.url?scp=85059004880&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85059004880&partnerID=8YFLogxK
U2 - 10.1145/3267809.3267832
DO - 10.1145/3267809.3267832
M3 - Conference contribution
T3 - SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing
SP - 249
EP - 262
BT - SoCC 2018 - Proceedings of the 2018 ACM Symposium on Cloud Computing
PB - Association for Computing Machinery, Inc
ER -