R-storm: Resource-aware scheduling in storm

Boyang Peng, Mohammad Hosseini, Zhihao Hong, Reza Farivar, Roy Campbell

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The era of big data has led to the emergence of new systems for real-time distributed stream processing, e.g., Apache Storm is one of the most popular stream processing systems in in- dustry today. However, Storm, like many other stream pro- cessing systems lacks an intelligent scheduling mechanism. The default round-robin scheduling currently deployed in Storm disregards resource demands and availability, and can therefore be inefficient at times. We present R-Storm (Resource-Aware Storm), a system that implements resource- aware scheduling within Storm. R-Storm is designed to in- crease overall throughput by maximizing resource utilization while minimizing network latency. When scheduling tasks, R-Storm can satisfy both soft and hard resource constraints as well as minimizing network distance between components that communicate with each other. We evaluate R-Storm on set of micro-benchmark Storm applications as well as Storm applications used in production at Yahoo! Inc. From our experimental results we conclude that R-Storm achieves 30-47% higher throughput and 69-350% better CPU utiliza- tion than default Storm for the micro-benchmarks. For the Yahoo! Storm applications, R-Storm outperforms default Storm by around 50% based on overall throughput. We also demonstrate that R-Storm performs much better when scheduling multiple Storm applications than default Storm.

Original languageEnglish (US)
Title of host publicationMiddleware 2015 - Proceedings of the 16th Annual Middleware Conference
PublisherAssociation for Computing Machinery, Inc
Pages149-161
Number of pages13
ISBN (Electronic)9781450336185
DOIs
StatePublished - Nov 24 2015
Event16th International Middleware Conference, Middleware 2015 - Vancouver, Canada
Duration: Dec 7 2015Dec 11 2015

Publication series

NameMiddleware 2015 - Proceedings of the 16th Annual Middleware Conference

Other

Other16th International Middleware Conference, Middleware 2015
CountryCanada
CityVancouver
Period12/7/1512/11/15

    Fingerprint

Keywords

  • Resource-aware scheduling
  • Storm
  • Stream

ASJC Scopus subject areas

  • Information Systems
  • Software

Cite this

Peng, B., Hosseini, M., Hong, Z., Farivar, R., & Campbell, R. (2015). R-storm: Resource-aware scheduling in storm. In Middleware 2015 - Proceedings of the 16th Annual Middleware Conference (pp. 149-161). (Middleware 2015 - Proceedings of the 16th Annual Middleware Conference). Association for Computing Machinery, Inc. https://doi.org/10.1145/2814576.2814808