Empirically revisiting and enhancing IR-based test-case prioritization

Qianyang Peng, August Shi, Lingming Zhang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Test-case prioritization (TCP) aims to detect regression bugs faster via reordering the tests run. While TCP has been studied for over 20 years, it was almost always evaluated using seeded faults/mutants as opposed to using real test failures. In this work, we study the recent change-aware information retrieval (IR) technique for TCP. Prior work has shown it performing better than traditional coverage-based TCP techniques, but it was only evaluated on a small-scale dataset with a cost-unaware metric based on seeded faults/mutants. We extend the prior work by conducting a much larger and more realistic evaluation as well as proposing enhancements that substantially improve the performance. In particular, we evaluate the original technique on a large-scale, real-world software-evolution dataset with real failures using both cost-aware and cost-unaware metrics under various configurations. Also, we design and evaluate hybrid techniques combining the IR features, historical test execution time, and test failure frequencies. Our results show that the change-aware IR technique outperforms stateof-the-art coverage-based techniques in this real-world setting, and our hybrid techniques improve even further upon the original IR technique. Moreover, we show that flaky tests have a substantial impact on evaluating the change-aware TCP techniques based on real test failures.

Original languageEnglish (US)
Title of host publicationISSTA 2020 - Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis
EditorsSarfraz Khurshid, Corina S. Pasareanu
PublisherAssociation for Computing Machinery
Pages324-336
Number of pages13
ISBN (Electronic)9781450380089
DOIs
StatePublished - Jul 18 2020
Externally publishedYes
Event29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020 - Virtual, Online, United States
Duration: Jul 18 2020Jul 22 2020

Publication series

NameISSTA 2020 - Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis

Conference

Conference29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020
Country/TerritoryUnited States
CityVirtual, Online
Period7/18/207/22/20

Keywords

  • Test-case prioritization
  • continuous integration
  • information retrieval

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'Empirically revisiting and enhancing IR-based test-case prioritization'. Together they form a unique fingerprint.

Cite this