Understanding vicious cycles in server clusters

Mohammad Maifi Hasan Khan, Shen Li, Jin Heo, Tarek Abdelzaher

Research output: Chapter in Book/Report/Conference proceedingConference contribution


In this paper, we present an automated on-line service for troubleshooting performance problems in server clusters caused by unintended vicious cycles. The tool complements a large volume of prior performance troubleshooting and diagnostic literature for server farms that identifies problems arising due to resource bottlenecks or failed components. We show that unintended interactions between components in large-scale systems can cause performance problems even in the absence of bottlenecks or failures. Our tool leverages discriminative sequence mining to identify anomalous sequences of events that are candidates for blame for the performance problem. The tool looks for patterns consistent with "vicious cycles" or unstable behavior, as such patterns, when present, are most likely to be problematic. It highlights candidates that are semantically conflicting, such as those arising when different performance management mechanisms make adjustments in conflicting directions. Our approach offers two key advantages in performance troubleshooting. First, it does not require detailed prior knowledge of the underlying system to diagnose the problem. Second, contrary to simple statistical techniques, such as correlation analysis, that work well for continuous variables, our scheme can also identify chains of events (labels) that may explain the root cause of a problem. Our service is deployed on a web server testbed of 17 machines. To make the comparison of our scheme to prior work more concrete, we first reproduce two real-life problem scenarios reported in earlier literature, then explore a third, new case study. In all cases, our tool reports the patterns that explain the cause of the problem without requiring detailed a priori knowledge.

Original languageEnglish (US)
Title of host publicationProceedings - 31st International Conference on Distributed Computing Systems, ICDCS 2011
Number of pages10
StatePublished - 2011
Event31st International Conference on Distributed Computing Systems, ICDCS 2011 - Minneapolis, MN, United States
Duration: Jun 20 2011Jul 24 2011

Publication series

NameProceedings - International Conference on Distributed Computing Systems


Other31st International Conference on Distributed Computing Systems, ICDCS 2011
Country/TerritoryUnited States
CityMinneapolis, MN


  • Adaptive components
  • Data center
  • Interactive complexity
  • Performance troubleshooting

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications


Dive into the research topics of 'Understanding vicious cycles in server clusters'. Together they form a unique fingerprint.

Cite this