Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests

Anjiang Wei, Pu Yi, Tao Xie, Darko Marinov, Wing Lam

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Software developers frequently check their code changes by running a set of tests against their code. Tests that can nondeterministi-cally pass or fail when run on the same code version are called flaky tests. These tests are a major problem because they can mislead developers to debug their recent code changes when the failures are unrelated to these changes. One prominent category of flaky tests is order-dependent (OD) tests, which can deterministically pass or fail depending on the order in which the set of tests are run. By detecting OD tests in advance, developers can fix these tests before they change their code. Due to the high cost required to explore all possible orders (n! permutations for n tests), prior work has developed tools that randomize orders to detect OD tests. Experiments have shown that randomization can detect many OD tests, and that most OD tests depend on just one other test to fail. However, there was no analysis of the probability that randomized orders detect OD tests. In this paper, we present the first such analysis and also present a simple change for sampling random test orders to increase the probability. We finally present a novel algorithm to systematically explore all consecutive pairs of tests, guaranteeing to detect all OD tests that depend on one other test, while running substantially fewer orders and tests than simply running all test pairs.

Original languageEnglish (US)
Title of host publicationTools and Algorithms for the Construction and Analysis of Systems - 27th International Conference, TACAS 2021 Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2021
EditorsJan Friso Groote, Kim Guldstrand Larsen
PublisherSpringer
Pages270-287
Number of pages18
ISBN (Print)9783030720155
DOIs
StatePublished - 2021
Event27th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2021 Held as Part of 24th European Joint Conferences on Theory and Practice of Software, ETAPS 2021 - Virtual, Online
Duration: Mar 27 2021Apr 1 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12651 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference27th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2021 Held as Part of 24th European Joint Conferences on Theory and Practice of Software, ETAPS 2021
CityVirtual, Online
Period3/27/214/1/21

Keywords

  • Flaky tests
  • Order dependent
  • Test-pair coverage

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests'. Together they form a unique fingerprint.

Cite this