Stela: Enabling stream processing systems to scale-in and scale-out on-demand

Le Xu; Boyang Peng; Indranil Gupta

doi:10.1109/IC2E.2016.38

Stela: Enabling stream processing systems to scale-in and scale-out on-demand

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The era of big data has led to the emergence of new real-time distributed stream processing engines like Apache Storm. We present Stela (STream processing ELAsticity), a stream processing system that supports scale-out and scale-in operations in an on-demand manner, i.e., when the user requests such a scaling operation. Stela meets two goals: 1) it optimizes post-scaling throughput, and 2) it minimizes interruption to the ongoing computation while the scaling operation is being carried out. We have integrated Stela into Apache Storm. We present experimental results using micro-benchmark Storm applications, as well as production applications from industry (Yahoo! Inc. and IBM). Our experiments show that compared to Apache Storm's default scheduler, Stela's scale-out operation achieves throughput that is 21-120% higher, and interruption time that is significantly smaller. Stela's scale-in operation chooses the right set of servers to remove and achieves 2X-5X higher throughput than Storm's default strategy.

Original language	English (US)
Title of host publication	Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016
Subtitle of host publication	Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	22-31
Number of pages	10
ISBN (Electronic)	9781509019618
DOIs	https://doi.org/10.1109/IC2E.2016.38
State	Published - Jun 1 2016
Event	4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016 - Berlin, Germany Duration: Apr 4 2016 → Apr 8 2016

Publication series

Name	Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016

Other

Other	4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016
Country	Germany
City	Berlin
Period	4/4/16 → 4/8/16

Keywords

Distributed Systems
Elasticity
Scalability
Stream Processing

ASJC Scopus subject areas

Control and Systems Engineering
Computer Networks and Communications

Access to Document

10.1109/IC2E.2016.38

Fingerprint Dive into the research topics of 'Stela: Enabling stream processing systems to scale-in and scale-out on-demand'. Together they form a unique fingerprint.

Cite this

Xu, L., Peng, B., & Gupta, I. (2016). Stela: Enabling stream processing systems to scale-in and scale-out on-demand. In Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016 (pp. 22-31). [7484160] (Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IC2E.2016.38

Stela : Enabling stream processing systems to scale-in and scale-out on-demand. / Xu, Le; Peng, Boyang; Gupta, Indranil.

Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016. Institute of Electrical and Electronics Engineers Inc., 2016. p. 22-31 7484160 (Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Xu, L, Peng, B & Gupta, I 2016, Stela: Enabling stream processing systems to scale-in and scale-out on-demand. in Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016., 7484160, Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016, Institute of Electrical and Electronics Engineers Inc., pp. 22-31, 4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016, Berlin, Germany, 4/4/16. https://doi.org/10.1109/IC2E.2016.38

Xu L, Peng B, Gupta I. Stela: Enabling stream processing systems to scale-in and scale-out on-demand. In Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016. Institute of Electrical and Electronics Engineers Inc. 2016. p. 22-31. 7484160. (Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016). https://doi.org/10.1109/IC2E.2016.38

Xu, Le ; Peng, Boyang ; Gupta, Indranil. / Stela : Enabling stream processing systems to scale-in and scale-out on-demand. Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 22-31 (Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016).

@inproceedings{cacd4aa67ffa460491acce31c8d3cf59,

title = "Stela: Enabling stream processing systems to scale-in and scale-out on-demand",

abstract = "The era of big data has led to the emergence of new real-time distributed stream processing engines like Apache Storm. We present Stela (STream processing ELAsticity), a stream processing system that supports scale-out and scale-in operations in an on-demand manner, i.e., when the user requests such a scaling operation. Stela meets two goals: 1) it optimizes post-scaling throughput, and 2) it minimizes interruption to the ongoing computation while the scaling operation is being carried out. We have integrated Stela into Apache Storm. We present experimental results using micro-benchmark Storm applications, as well as production applications from industry (Yahoo! Inc. and IBM). Our experiments show that compared to Apache Storm's default scheduler, Stela's scale-out operation achieves throughput that is 21-120% higher, and interruption time that is significantly smaller. Stela's scale-in operation chooses the right set of servers to remove and achieves 2X-5X higher throughput than Storm's default strategy.",

keywords = "Distributed Systems, Elasticity, Scalability, Stream Processing",

author = "Le Xu and Boyang Peng and Indranil Gupta",

note = "Funding Information: We thank our collaborators at Yahoo! Inc. for providing us the Storm topologies: Matt Ahrens, Bobby Evans, Derek Dagit, and the whole Storm development team at Yahoo. This work was supported in part by the following grants: NSF CNS 1409416, NSF CNS 1319527, NSF CCF 0964471, and AFOSR/AFRL FA8750-11-2-0084, and a generous gift from Microsoft. Publisher Copyright: {\textcopyright} 2016 IEEE. Copyright: Copyright 2016 Elsevier B.V., All rights reserved.; 4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016 ; Conference date: 04-04-2016 Through 08-04-2016",

year = "2016",

month = jun,

day = "1",

doi = "10.1109/IC2E.2016.38",

language = "English (US)",

series = "Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "22--31",

booktitle = "Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016",

address = "United States",

}

TY - GEN

T1 - Stela

T2 - 4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016

AU - Xu, Le

AU - Peng, Boyang

AU - Gupta, Indranil

N1 - Funding Information: We thank our collaborators at Yahoo! Inc. for providing us the Storm topologies: Matt Ahrens, Bobby Evans, Derek Dagit, and the whole Storm development team at Yahoo. This work was supported in part by the following grants: NSF CNS 1409416, NSF CNS 1319527, NSF CCF 0964471, and AFOSR/AFRL FA8750-11-2-0084, and a generous gift from Microsoft. Publisher Copyright: © 2016 IEEE. Copyright: Copyright 2016 Elsevier B.V., All rights reserved.

PY - 2016/6/1

Y1 - 2016/6/1

N2 - The era of big data has led to the emergence of new real-time distributed stream processing engines like Apache Storm. We present Stela (STream processing ELAsticity), a stream processing system that supports scale-out and scale-in operations in an on-demand manner, i.e., when the user requests such a scaling operation. Stela meets two goals: 1) it optimizes post-scaling throughput, and 2) it minimizes interruption to the ongoing computation while the scaling operation is being carried out. We have integrated Stela into Apache Storm. We present experimental results using micro-benchmark Storm applications, as well as production applications from industry (Yahoo! Inc. and IBM). Our experiments show that compared to Apache Storm's default scheduler, Stela's scale-out operation achieves throughput that is 21-120% higher, and interruption time that is significantly smaller. Stela's scale-in operation chooses the right set of servers to remove and achieves 2X-5X higher throughput than Storm's default strategy.

AB - The era of big data has led to the emergence of new real-time distributed stream processing engines like Apache Storm. We present Stela (STream processing ELAsticity), a stream processing system that supports scale-out and scale-in operations in an on-demand manner, i.e., when the user requests such a scaling operation. Stela meets two goals: 1) it optimizes post-scaling throughput, and 2) it minimizes interruption to the ongoing computation while the scaling operation is being carried out. We have integrated Stela into Apache Storm. We present experimental results using micro-benchmark Storm applications, as well as production applications from industry (Yahoo! Inc. and IBM). Our experiments show that compared to Apache Storm's default scheduler, Stela's scale-out operation achieves throughput that is 21-120% higher, and interruption time that is significantly smaller. Stela's scale-in operation chooses the right set of servers to remove and achieves 2X-5X higher throughput than Storm's default strategy.

KW - Distributed Systems

KW - Elasticity

KW - Scalability

KW - Stream Processing

UR - http://www.scopus.com/inward/record.url?scp=84978062188&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84978062188&partnerID=8YFLogxK

U2 - 10.1109/IC2E.2016.38

DO - 10.1109/IC2E.2016.38

M3 - Conference contribution

AN - SCOPUS:84978062188

T3 - Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016

SP - 22

EP - 31

BT - Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 4 April 2016 through 8 April 2016

ER -