Failure Diagnosis for Distributed Systems Using Targeted Fault Injection

Cuong Pham, Long Wang, Byung Chul Tak, Salman Baset, Chunqiang Tang, Zbigniew Kalbarczyk, Ravishankar K. Iyer

Research output: Contribution to journalArticlepeer-review

Abstract

This paper introduces a novel approach to automating failure diagnostics in distributed systems by combining fault injection and data analytics. We use fault injection to populate the database of failures for a target distributed system. When a failure is reported from production environment, the database is queried to find 'matched' failures generated by fault injections. Relying on the assumption that similar faults generate similar failures, we use information from the matched failures as hints to locate the actual root cause of the reported failures. In order to implement this approach, we introduce techniques for (i) reconstructing end-to-end execution flows of distributed software components, (ii) computing the similarity of the reconstructed flows, and (iii) performing precise fault injection at pre-specified executing points in distributed systems. We have evaluated our approach using an OpenStack cloud platform, a popular cloud infrastructure management system. Our experimental results showed that this approach is effective in determining the root causes, e.g., fault types and affected components, for 71-100 percent of tested failures. Furthermore, it can provide fault locations close to actual ones and can easily be used to find and fix actual root causes. We have also validated this technique by localizing real bugs that occurred in OpenStack.

Original languageEnglish (US)
Article number7484300
Pages (from-to)503-516
Number of pages14
JournalIEEE Transactions on Parallel and Distributed Systems
Volume28
Issue number2
DOIs
StatePublished - Feb 1 2017

Keywords

  • Fault injection
  • distributed system
  • failure diagnosis
  • fault localization
  • processing flow

ASJC Scopus subject areas

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'Failure Diagnosis for Distributed Systems Using Targeted Fault Injection'. Together they form a unique fingerprint.

Cite this