TY - JOUR
T1 - How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub Projects
AU - Luo, Qi
AU - Moran, Kevin
AU - Zhang, Lingming
AU - Poshyvanyk, Denys
N1 - Funding Information:
We would like thank the anonymous TSE and FSE reviewers for their insightful comments that significantly improved the quality of a previous version of this paper and provided useful suggestions for extending the study as presented in this journal paper submission. This work is supported in part by the NSF CCF-1218129, CCF-1566589, and NSF CNS-1510239 grants. Any opinions, findings, and conclusions expressed herein are the authors’ and do not necessarily reflect those of the sponsors. This paper is an extension of “A Large-scale Empirical Comparison of Static and Dynamic Test Case Prioritization Techniques” that appeared in Proceedings of 24th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE’16), Seattle, WA, November 13-18, 2016, pp. 559-570.
Publisher Copyright:
© 1976-2012 IEEE.
PY - 2019/11/1
Y1 - 2019/11/1
N2 - Test Case Prioritization (TCP) is an increasingly important regression testing technique for reordering test cases according to a pre-defined goal, particularly as agile practices gain adoption. To better understand these techniques, we perform the first extensive study aimed at empirically evaluating four static TCP techniques, comparing them with state-of-research dynamic TCP techniques across several quality metrics. This study was performed on 58 real-word Java programs encompassing 714 KLoC and results in several notable observations. First, our results across two effectiveness metrics (the Average Percentage of Faults Detected APFD and the cost cognizant APFDc) illustrate that at test-class granularity, these metrics tend to correlate, but this correlation does not hold at test-method granularity. Second, our analysis shows that static techniques can be surprisingly effective, particularly when measured by APFDc. Third, we found that TCP techniques tend to perform better on larger programs, but that program size does not affect comparative performance measures between techniques. Fourth, software evolution does not significantly impact comparative performance results between TCP techniques. Fifth, neither the number nor type of mutants utilized dramatically impact measures of TCP effectiveness under typical experimental settings. Finally, our similarity analysis illustrates that highly prioritized test cases tend to uncover dissimilar faults.
AB - Test Case Prioritization (TCP) is an increasingly important regression testing technique for reordering test cases according to a pre-defined goal, particularly as agile practices gain adoption. To better understand these techniques, we perform the first extensive study aimed at empirically evaluating four static TCP techniques, comparing them with state-of-research dynamic TCP techniques across several quality metrics. This study was performed on 58 real-word Java programs encompassing 714 KLoC and results in several notable observations. First, our results across two effectiveness metrics (the Average Percentage of Faults Detected APFD and the cost cognizant APFDc) illustrate that at test-class granularity, these metrics tend to correlate, but this correlation does not hold at test-method granularity. Second, our analysis shows that static techniques can be surprisingly effective, particularly when measured by APFDc. Third, we found that TCP techniques tend to perform better on larger programs, but that program size does not affect comparative performance measures between techniques. Fourth, software evolution does not significantly impact comparative performance results between TCP techniques. Fifth, neither the number nor type of mutants utilized dramatically impact measures of TCP effectiveness under typical experimental settings. Finally, our similarity analysis illustrates that highly prioritized test cases tend to uncover dissimilar faults.
KW - Regression testing
KW - dynamic
KW - mutation analysis
KW - static
KW - test case prioritization
UR - http://www.scopus.com/inward/record.url?scp=85044789352&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85044789352&partnerID=8YFLogxK
U2 - 10.1109/TSE.2018.2822270
DO - 10.1109/TSE.2018.2822270
M3 - Article
AN - SCOPUS:85044789352
SN - 0098-5589
VL - 45
SP - 1054
EP - 1080
JO - IEEE Transactions on Software Engineering
JF - IEEE Transactions on Software Engineering
IS - 11
M1 - 8329518
ER -