Dependent-test-aware regression testing techniques

Wing Lam, August Shi, Reed Oei, Sai Zhang, Michael D. Ernst, Tao Xie

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Developers typically rely on regression testing techniques to ensure that their changes do not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same version of code and tests. One prominent type of flaky tests is order-dependent (OD) tests, which are tests that pass when run in one order but fail when run in another order. Although OD tests may cause flaky-test failures, OD tests can help developers run their tests faster by allowing them to share resources. We propose to make regression testing techniques dependent-test-aware to reduce flaky-test failures. To understand the necessity of dependent-test-aware regression testing techniques, we conduct the first study on the impact of OD tests on three regression testing techniques: test prioritization, test selection, and test parallelization. In particular, we implement 4 test prioritization, 6 test selection, and 2 test parallelization algorithms, and we evaluate them on 11 Java modules with OD tests. When we run the orders produced by the traditional, dependent-test-unaware regression testing algorithms, 82% of human-written test suites and 100% of automatically-generated test suites with OD tests have at least one flaky-test failure. We develop a general approach for enhancing regression testing algorithms to make them dependent-test-aware, and apply our approach to 12 algorithms. Compared to traditional, unenhanced regression testing algorithms, the enhanced algorithms use provided test dependencies to produce orders with different permutations or extra tests. Our evaluation shows that, in comparison to the orders produced by unenhanced algorithms, the orders produced by enhanced algorithms (1) have overall 80% fewer flaky-test failures due to OD tests, and (2) may add extra tests but run only 1% slower on average. Our results suggest that enhancing regression testing algorithms to be dependent-test-aware can substantially reduce flaky-test failures with only a minor slowdown to run the tests.

Original languageEnglish (US)
Title of host publicationISSTA 2020 - Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis
EditorsSarfraz Khurshid, Corina S. Pasareanu
PublisherAssociation for Computing Machinery, Inc
Pages298-311
Number of pages14
ISBN (Electronic)9781450380089
DOIs
StatePublished - Jul 18 2020
Event29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020 - Virtual, Online, United States
Duration: Jul 18 2020Jul 22 2020

Publication series

NameISSTA 2020 - Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis

Conference

Conference29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020
CountryUnited States
CityVirtual, Online
Period7/18/207/22/20

Keywords

  • flaky test
  • order-dependent test
  • regression testing

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Dependent-test-aware regression testing techniques'. Together they form a unique fingerprint.

Cite this