Evaluating test-suite reduction in real software evolution

August Shi, Alex Gyori, Suleman Mahmood, Peiyuan Zhao, Darko Marinov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Test-suite reduction (TSR) speeds up regression testing by removing redundant tests from the test suite, thus running fewer tests in the future builds. To decide whether to use TSR or not, a developer needs some way to predict howwell the reduced test suite will detect real faults in the future compared to the original test suite. Prior research evaluated the cost of TSR using only program versions with seeded faults, but such evaluations do not explicitly predict the effectiveness of the reduced test suite in future builds. We perform the first extensive study of TSR using real test failures in (failed) builds that occurred for real code changes. We analyze 1478 failed builds from 32 GitHub projects that run their tests on Travis. Each failed build can have multiple faults, so we propose a family of mappings from test failures to faults. We use these mappings to compute Failed-Build Detection Loss (FBDL), the percentage of failed builds where the reduced test suite misses to detect all the faults detected by the original test suite. We find that FBDL can be up to 52.2%, which is higher than suggested by traditional TSR metrics. Moreover, traditional TSR metrics are not good predictors of FBDL, making it difficult for developers to decide whether to use reduced test suites.

Original languageEnglish (US)
Title of host publicationISSTA 2018 - Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis
EditorsEric Bodden, Frank Tip
PublisherAssociation for Computing Machinery, Inc
Pages84-94
Number of pages11
ISBN (Electronic)9781450356992
DOIs
StatePublished - Jul 12 2018
Event27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018 - Amsterdam, Netherlands
Duration: Jul 16 2018Jul 21 2018

Publication series

NameISSTA 2018 - Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis

Other

Other27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2018
CountryNetherlands
CityAmsterdam
Period7/16/187/21/18

Keywords

  • Continuous integration
  • Regression testing
  • Test-suite reduction

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Software
  • Computer Science Applications

Fingerprint Dive into the research topics of 'Evaluating test-suite reduction in real software evolution'. Together they form a unique fingerprint.

  • Cite this

    Shi, A., Gyori, A., Mahmood, S., Zhao, P., & Marinov, D. (2018). Evaluating test-suite reduction in real software evolution. In E. Bodden, & F. Tip (Eds.), ISSTA 2018 - Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 84-94). (ISSTA 2018 - Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis). Association for Computing Machinery, Inc. https://doi.org/10.1145/3213846.3213875