TY - GEN
T1 - Empirically revisiting and enhancing IR-based test-case prioritization
AU - Peng, Qianyang
AU - Shi, August
AU - Zhang, Lingming
N1 - Funding Information:
We thank Wing Lam, Owolabi Legunsen, and Darko Marinov for their extensive discussion with us on this work. We acknowledge support for research on test quality from Futurewei. We also ac-knowlege support from Alibaba. This work was partially supported by NSF grants CNS-1646305, CCF-1763906, and OAC-1839010.
Publisher Copyright:
© 2020 ACM.
PY - 2020/7/18
Y1 - 2020/7/18
N2 - Test-case prioritization (TCP) aims to detect regression bugs faster via reordering the tests run. While TCP has been studied for over 20 years, it was almost always evaluated using seeded faults/mutants as opposed to using real test failures. In this work, we study the recent change-aware information retrieval (IR) technique for TCP. Prior work has shown it performing better than traditional coverage-based TCP techniques, but it was only evaluated on a small-scale dataset with a cost-unaware metric based on seeded faults/mutants. We extend the prior work by conducting a much larger and more realistic evaluation as well as proposing enhancements that substantially improve the performance. In particular, we evaluate the original technique on a large-scale, real-world software-evolution dataset with real failures using both cost-aware and cost-unaware metrics under various configurations. Also, we design and evaluate hybrid techniques combining the IR features, historical test execution time, and test failure frequencies. Our results show that the change-aware IR technique outperforms stateof-the-art coverage-based techniques in this real-world setting, and our hybrid techniques improve even further upon the original IR technique. Moreover, we show that flaky tests have a substantial impact on evaluating the change-aware TCP techniques based on real test failures.
AB - Test-case prioritization (TCP) aims to detect regression bugs faster via reordering the tests run. While TCP has been studied for over 20 years, it was almost always evaluated using seeded faults/mutants as opposed to using real test failures. In this work, we study the recent change-aware information retrieval (IR) technique for TCP. Prior work has shown it performing better than traditional coverage-based TCP techniques, but it was only evaluated on a small-scale dataset with a cost-unaware metric based on seeded faults/mutants. We extend the prior work by conducting a much larger and more realistic evaluation as well as proposing enhancements that substantially improve the performance. In particular, we evaluate the original technique on a large-scale, real-world software-evolution dataset with real failures using both cost-aware and cost-unaware metrics under various configurations. Also, we design and evaluate hybrid techniques combining the IR features, historical test execution time, and test failure frequencies. Our results show that the change-aware IR technique outperforms stateof-the-art coverage-based techniques in this real-world setting, and our hybrid techniques improve even further upon the original IR technique. Moreover, we show that flaky tests have a substantial impact on evaluating the change-aware TCP techniques based on real test failures.
KW - Test-case prioritization
KW - continuous integration
KW - information retrieval
UR - http://www.scopus.com/inward/record.url?scp=85088916280&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85088916280&partnerID=8YFLogxK
U2 - 10.1145/3395363.3397383
DO - 10.1145/3395363.3397383
M3 - Conference contribution
AN - SCOPUS:85088916280
T3 - ISSTA 2020 - Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis
SP - 324
EP - 336
BT - ISSTA 2020 - Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis
A2 - Khurshid, Sarfraz
A2 - Pasareanu, Corina S.
PB - Association for Computing Machinery
T2 - 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020
Y2 - 18 July 2020 through 22 July 2020
ER -