A Large-Scale Longitudinal Study of Flaky Tests

Wing Lam, Stefan Winter, Anjiang Wei, Tao Xie, Darko Marinov, Jonathan Bell

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Flaky tests that non-deterministically pass or fail without any code changes constitute an impediment to regression testing. To understand when and how flaky tests can be detected most efficiently, we analyzed the commit histories of known flaky tests. We find that 75% of flaky tests are flaky when added, indicating substantial value for developers to run detectors specifically on newly added tests. The percentage of flaky tests that can be detected early increases to 85% when detectors are run on both newly added and directly modified tests.

Original languageEnglish (US)
Title of host publicationSoftware Engineering 2022 - Fachtagung des GI-Fachbereichs Softwaretechnik
EditorsLars Grunske, Janet Siegmund, Andreas Vogelsang
PublisherGesellschaft fur Informatik (GI)
Pages57-59
Number of pages3
ISBN (Electronic)9783885797142
StatePublished - 2022
EventFachtagung des GI-Fachbereichs Softwaretechnik, Software Engineering 2022 - Conference of the GI Software Engineering Section, Software Engineering 2022 - Virtual, Online, Germany
Duration: Feb 21 2022Feb 25 2022

Publication series

NameLecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI)
VolumeP-320
ISSN (Print)1617-5468
ISSN (Electronic)2944-7682

Conference

ConferenceFachtagung des GI-Fachbereichs Softwaretechnik, Software Engineering 2022 - Conference of the GI Software Engineering Section, Software Engineering 2022
Country/TerritoryGermany
CityVirtual, Online
Period2/21/222/25/22

Keywords

  • Flaky Tests
  • Regression Testing

ASJC Scopus subject areas

  • Computer Science Applications

Fingerprint

Dive into the research topics of 'A Large-Scale Longitudinal Study of Flaky Tests'. Together they form a unique fingerprint.

Cite this