DeFlaker: Automatically Detecting Flaky Tests

Jonathan Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, Darko Marinov

Research output: Contribution to journalConference articlepeer-review

Abstract

Developers often run tests to check that their latest changes to a code repository did not break any previously working functionality. Ideally, any new test failures would indicate regressions caused by the latest changes. However, some test failures may not be due to the latest changes but due to non-determinism in the tests, popularly called flaky tests. The typical way to detect flaky tests is to rerun failing tests repeatedly. Unfortunately, rerunning failing tests can be costly and can slow down the development cycle. We present the first extensive evaluation of rerunning failing tests and propose a new technique, called DeFlaker, that detects if a test failure is due to a flaky test without rerunning and with very low runtime overhead. DeFlaker monitors the coverage of latest code changes and marks as flaky any newly failing test that did not execute any of the changes. We deployed DeFlaker live, in the build process of 96 Java projects on TravisCI, and found 87 previously unknown flaky tests in 10 of these projects. We also ran experiments on project histories, where DeFlaker detected 1, 874 flaky tests from 4, 846 failures, with a low false alarm rate (1.5%). DeFlaker had a higher recall (95.5% vs. 23%) of confirmed flaky tests than Maven’s default flaky test detector.

Original languageEnglish (US)
Pages (from-to)433-444
Number of pages12
JournalProceedings - International Conference on Software Engineering
Volume2018-January
DOIs
StatePublished - 2018
Event40th International Conference on Software Engineering, ICSE 2018 - Gothenburg, Sweden
Duration: May 27 2018Jun 3 2018

Keywords

  • Software testing
  • code coverage
  • flaky tests

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'DeFlaker: Automatically Detecting Flaky Tests'. Together they form a unique fingerprint.

Cite this