Modeling probabilistic measurement correlations for problem determination in large-scale distributed systems

Jing Gao, Guofei Jiang, Haifeng Chen, Jiawei Han

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the growing complexity in computer systems, it has been a real challenge to detect and diagnose problems in today's large-scale distributed systems. Usually, the correlations between measurements collected across the distributed system contain rich information about the system behaviors, and thus a reasonable model to describe such correlations is crucially important in detecting and locating system problems. In this paper, we propose a transition probability model based on markov properties to characterize pairwise measurement correlations. The proposed method can discover both the spatial (across system measurements) and temporal (across observation time) correlations, and thus such a model can successfully represent the system normal profiles. Problem determination and localization under this framework is fast and convenient. The framework is general enough to discover any types of correlations (e.g. linear or non-linear). Also, model updating, system problem detection and diagnosis can be conducted effectively and efficiently. Experimental results show that, the proposed method can detect the anomalous events and locate the problematic sources by analyzing the real monitoring data collected from three companies' infrastructures.

Original languageEnglish (US)
Title of host publication2009 29th IEEE International Conference on Distributed Computing Systems Workshops, ICDCS, 09
Pages623-630
Number of pages8
DOIs
StatePublished - 2009
Event2009 29th IEEE International Conference on Distributed Computing Systems Workshops, ICDCS, 09 - Montreal, QC, Canada
Duration: Jun 22 2009Jun 26 2009

Other

Other2009 29th IEEE International Conference on Distributed Computing Systems Workshops, ICDCS, 09
Country/TerritoryCanada
CityMontreal, QC
Period6/22/096/26/09

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'Modeling probabilistic measurement correlations for problem determination in large-scale distributed systems'. Together they form a unique fingerprint.

Cite this