Understanding reproducibility and characteristics of flaky tests through test reruns in java projects

Wing Lam, Stefan Winter, Angello Astorga, Victoria Stodden, Darko Marinov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Flaky tests are tests that can non-deterministically pass and fail. They pose a major impediment to regression testing, because they provide an inconclusive assessment on whether recent code changes contain faults or not. Prior studies of flaky tests have proposed tools to detect flaky tests and identified various sources of flakiness in tests, e.g., order-dependent (OD) tests that deterministically fail for some order of tests in a test suite but deterministically pass for some other orders. Several of these studies have focused on OD tests. We focus on an important and under-explored source of flakiness in tests: non-order-dependent tests that can nondeterministically pass and fail even for the same order of tests. Instead of using specialized tools that aim to detect flaky tests, we run tests using the tool configured by the developers. Specifically, we perform our empirical evaluation on Java projects that rely on the Maven Surefire plugin to run tests. We re-execute each test suite 4000 times, potentially in different test-class orders, and we label tests as flaky if our runs have both pass and fail outcomes across these reruns. We obtain a dataset of 107 flaky tests and study various characteristics of these tests. We find that many tests previously called "non-order-dependent"actually do depend on the order and can fail with very different failure rates for different orders.

Original languageEnglish (US)
Title of host publicationProceedings - 2020 IEEE 31st International Symposium on Software Reliability Engineering, ISSRE 2020
EditorsMarco Vieira, Henrique Madeira, Nuno Antunes, Zheng Zheng
PublisherIEEE Computer Society
Pages403-413
Number of pages11
ISBN (Electronic)9781728198705
DOIs
StatePublished - Oct 2020
Event31st IEEE International Symposium on Software Reliability Engineering, ISSRE 2020 - Virtual, Coimbra, Portugal
Duration: Oct 12 2020Oct 15 2020

Publication series

NameProceedings - International Symposium on Software Reliability Engineering, ISSRE
Volume2020-October
ISSN (Print)1071-9458

Conference

Conference31st IEEE International Symposium on Software Reliability Engineering, ISSRE 2020
CountryPortugal
CityVirtual, Coimbra
Period10/12/2010/15/20

Keywords

  • Flaky tests
  • Regression testing
  • Reproducibility

ASJC Scopus subject areas

  • Software
  • Safety, Risk, Reliability and Quality

Fingerprint Dive into the research topics of 'Understanding reproducibility and characteristics of flaky tests through test reruns in java projects'. Together they form a unique fingerprint.

Cite this