R-storm: Resource-aware scheduling in storm

Boyang Peng, Mohammad Hosseini, Zhihao Hong, Reza Farivar, Roy Campbell

Research output: Chapter in Book/Report/Conference proceedingConference contribution


The era of big data has led to the emergence of new systems for real-time distributed stream processing, e.g., Apache Storm is one of the most popular stream processing systems in in- dustry today. However, Storm, like many other stream pro- cessing systems lacks an intelligent scheduling mechanism. The default round-robin scheduling currently deployed in Storm disregards resource demands and availability, and can therefore be inefficient at times. We present R-Storm (Resource-Aware Storm), a system that implements resource- aware scheduling within Storm. R-Storm is designed to in- crease overall throughput by maximizing resource utilization while minimizing network latency. When scheduling tasks, R-Storm can satisfy both soft and hard resource constraints as well as minimizing network distance between components that communicate with each other. We evaluate R-Storm on set of micro-benchmark Storm applications as well as Storm applications used in production at Yahoo! Inc. From our experimental results we conclude that R-Storm achieves 30-47% higher throughput and 69-350% better CPU utiliza- tion than default Storm for the micro-benchmarks. For the Yahoo! Storm applications, R-Storm outperforms default Storm by around 50% based on overall throughput. We also demonstrate that R-Storm performs much better when scheduling multiple Storm applications than default Storm.

Original languageEnglish (US)
Title of host publicationMiddleware 2015 - Proceedings of the 16th Annual Middleware Conference
PublisherAssociation for Computing Machinery, Inc
Number of pages13
ISBN (Electronic)9781450336185
StatePublished - Nov 24 2015
Event16th International Middleware Conference, Middleware 2015 - Vancouver, Canada
Duration: Dec 7 2015Dec 11 2015

Publication series

NameMiddleware 2015 - Proceedings of the 16th Annual Middleware Conference


Other16th International Middleware Conference, Middleware 2015


  • Resource-aware scheduling
  • Storm
  • Stream

ASJC Scopus subject areas

  • Information Systems
  • Software


Dive into the research topics of 'R-storm: Resource-aware scheduling in storm'. Together they form a unique fingerprint.

Cite this