Self-configuring information management for large-scale service overlays

Service overlay networks (SON) provide important infrastructure support for many emerging distributed applications such as web service composition, distributed stream processing, and workflow management. Quality-sensitive distributed applications such as multimedia services and on-line data analysis often desire the SON to provide up-to-date dynamic information about different overlay nodes and overlay links. However, it is a challenging task to provide scalable and efficient information management for large-scale SONs, where both system conditions and application requirements can change over time. In this paper, we present InfoEye, a model-based self-configuring distributed information management system that consists of a set of monitoring sensors deployed on different overlay nodes. InfoEye can dynamically configure the operations of different sensors based on current statistical application query patterns and system attribute distributions. Thus, InfoEye can greatly improve the scalability of SON by answering information queries with minimum monitoring overhead. We have implemented a prototype of InfoEye and evaluated its performance using both extensive simulations and micro-benchmark experiments on PlanetLab. The experimental results show that InfoEye can significantly reduce the information management overhead compared with existing approaches. In addition, InfoEye can quickly reconfigure itself in response to application requirement and system information pattern changes.