Optimal component composition for scalable stream processing

Xiaohui Gu, Philip S. Yu, Klara Nahrstedt

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Stream processing has become increasingly important with emergence of stream applications such as audio/video surveillance, stock price tracing, and sensor data analysis. A challenging problem is to provide optimal component composition in a distributed stream processing environment. The goal of optimal component composition is to achieve load balancing subject to multiple function, resource, and quality-of-service (QoS) constraints while composing stream applications. In this paper, we present an adaptive composition probing (ACP) approach to the problem. Different from previous work, ACP provides a new hybrid approach that combines distributed composition probing with coarse-grain global state management. Guided by the coarse-grain global state information, ACP selectively probes a subset of candidate components to discover an approximately optimal component composition. Further, ACP is self-tuning, which can adaptively adjust the number of probes to maintain a specified composition performance target (i.e., composition success rate) in a dynamic stream environment. While the optimal component composition problem is NP-hard, our ACP approach provides an adaptive polynomial approximation solution. We have conducted extensive simulation experiments to show the efficiency, scalability, and adaptability of the ACP approach by comparing with other alternative solutions.

Original languageEnglish (US)
Title of host publicationProceedings - 25th IEEE International Conference on Distributed Computing Systems
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages773-782
Number of pages10
StatePublished - 2005
Event25th IEEE International Conference on Distributed Computing Systems - Columbus, OH, United States
Duration: Jun 6 2005Jun 10 2005

Other

Other25th IEEE International Conference on Distributed Computing Systems
CountryUnited States
CityColumbus, OH
Period6/6/056/10/05

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Optimal component composition for scalable stream processing'. Together they form a unique fingerprint.

Cite this