Abstract
The era of big data has led to the emergence of new real-time distributed stream processing engines like Apache Storm. We present Stela (STream processing ELAsticity), a stream processing system that supports scale-out and scale-in operations in an on-demand manner, i.e., when the user requests such a scaling operation. Stela meets two goals: 1) it optimizes post-scaling throughput, and 2) it minimizes interruption to the ongoing computation while the scaling operation is being carried out. We have integrated Stela into Apache Storm. We present experimental results using micro-benchmark Storm applications, as well as production applications from industry (Yahoo! Inc. and IBM). Our experiments show that compared to Apache Storm's default scheduler, Stela's scale-out operation achieves throughput that is 21-120% higher, and interruption time that is significantly smaller. Stela's scale-in operation chooses the right set of servers to remove and achieves 2X-5X higher throughput than Storm's default strategy.
| Original language | English (US) |
|---|---|
| Title of host publication | Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016 |
| Subtitle of host publication | Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| Pages | 22-31 |
| Number of pages | 10 |
| ISBN (Electronic) | 9781509019618 |
| DOIs | |
| State | Published - Jun 1 2016 |
| Event | 4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016 - Berlin, Germany Duration: Apr 4 2016 → Apr 8 2016 |
Publication series
| Name | Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016 |
|---|
Other
| Other | 4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016 |
|---|---|
| Country | Germany |
| City | Berlin |
| Period | 4/4/16 → 4/8/16 |
Fingerprint
Keywords
- Distributed Systems
- Elasticity
- Scalability
- Stream Processing
ASJC Scopus subject areas
- Control and Systems Engineering
- Computer Networks and Communications
Cite this
Stela : Enabling stream processing systems to scale-in and scale-out on-demand. / Xu, Le; Peng, Boyang; Gupta, Indranil.
Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016. Institute of Electrical and Electronics Engineers Inc., 2016. p. 22-31 7484160 (Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Stela
T2 - Enabling stream processing systems to scale-in and scale-out on-demand
AU - Xu, Le
AU - Peng, Boyang
AU - Gupta, Indranil
PY - 2016/6/1
Y1 - 2016/6/1
N2 - The era of big data has led to the emergence of new real-time distributed stream processing engines like Apache Storm. We present Stela (STream processing ELAsticity), a stream processing system that supports scale-out and scale-in operations in an on-demand manner, i.e., when the user requests such a scaling operation. Stela meets two goals: 1) it optimizes post-scaling throughput, and 2) it minimizes interruption to the ongoing computation while the scaling operation is being carried out. We have integrated Stela into Apache Storm. We present experimental results using micro-benchmark Storm applications, as well as production applications from industry (Yahoo! Inc. and IBM). Our experiments show that compared to Apache Storm's default scheduler, Stela's scale-out operation achieves throughput that is 21-120% higher, and interruption time that is significantly smaller. Stela's scale-in operation chooses the right set of servers to remove and achieves 2X-5X higher throughput than Storm's default strategy.
AB - The era of big data has led to the emergence of new real-time distributed stream processing engines like Apache Storm. We present Stela (STream processing ELAsticity), a stream processing system that supports scale-out and scale-in operations in an on-demand manner, i.e., when the user requests such a scaling operation. Stela meets two goals: 1) it optimizes post-scaling throughput, and 2) it minimizes interruption to the ongoing computation while the scaling operation is being carried out. We have integrated Stela into Apache Storm. We present experimental results using micro-benchmark Storm applications, as well as production applications from industry (Yahoo! Inc. and IBM). Our experiments show that compared to Apache Storm's default scheduler, Stela's scale-out operation achieves throughput that is 21-120% higher, and interruption time that is significantly smaller. Stela's scale-in operation chooses the right set of servers to remove and achieves 2X-5X higher throughput than Storm's default strategy.
KW - Distributed Systems
KW - Elasticity
KW - Scalability
KW - Stream Processing
UR - http://www.scopus.com/inward/record.url?scp=84978062188&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84978062188&partnerID=8YFLogxK
U2 - 10.1109/IC2E.2016.38
DO - 10.1109/IC2E.2016.38
M3 - Conference contribution
AN - SCOPUS:84978062188
T3 - Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016
SP - 22
EP - 31
BT - Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016
PB - Institute of Electrical and Electronics Engineers Inc.
ER -