TY - GEN
T1 - A Theoretical Analysis of Random Regression Test Prioritization
AU - Yi, Pu
AU - Wang, Hao
AU - Xie, Tao
AU - Marinov, Darko
AU - Lam, Wing
N1 - We thank Anjiang Wei, Dezhi Ran, and Sasa Misailovic for their help. This work was partially supported by US NSF grants CCF-1763788. CCF-1956374, NSFC grant No. 62161146003, Tencent Foundation, and XPLORER PRIZE. We acknowledge support for research on regression testing from Dragon Testing, Microsoft, and Qualcomm. Tao Xie is the corresponding author, and also affiliated with Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, China.
Acknowledgments. We thank Anjiang Wei, Dezhi Ran, and Sasa Misailovic for their help. This work was partially supported by US NSF grants CCF-1763788. CCF-1956374, NSFC grant No. 62161146003, Tencent Foundation, and XPLORER PRIZE. We acknowledge support for research on regression testing from Dragon Testing, Microsoft, and Qualcomm. Tao Xie is the corresponding author, and also affiliated with Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, China.
PY - 2022
Y1 - 2022
N2 - Regression testing is an important activity to check software changes by running the tests in a test suite to inform the developers whether the changes lead to test failures. Regression test prioritization (RTP) aims to inform the developers faster by ordering the test suite so that tests likely to fail are run earlier. Many RTP techniques have been proposed and are often compared with the random RTP baseline by sampling some of the n! different test-suite orders for a test suite with n tests. However, there is no theoretical analysis of random RTP. We present such an analysis, deriving probability mass functions and expected values for metrics and scenarios commonly used in RTP research. Using our analysis, we revisit some of the most highly cited RTP papers and find that some presented results may be due to insufficient sampling. Future RTP research can leverage our analysis and need not use random sampling but can use our simple formulas or algorithms to more precisely compare with random RTP.
AB - Regression testing is an important activity to check software changes by running the tests in a test suite to inform the developers whether the changes lead to test failures. Regression test prioritization (RTP) aims to inform the developers faster by ordering the test suite so that tests likely to fail are run earlier. Many RTP techniques have been proposed and are often compared with the random RTP baseline by sampling some of the n! different test-suite orders for a test suite with n tests. However, there is no theoretical analysis of random RTP. We present such an analysis, deriving probability mass functions and expected values for metrics and scenarios commonly used in RTP research. Using our analysis, we revisit some of the most highly cited RTP papers and find that some presented results may be due to insufficient sampling. Future RTP research can leverage our analysis and need not use random sampling but can use our simple formulas or algorithms to more precisely compare with random RTP.
KW - Analysis
KW - Random
KW - Regression Test Prioritization
UR - http://www.scopus.com/inward/record.url?scp=85128777909&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85128777909&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-99527-0_12
DO - 10.1007/978-3-030-99527-0_12
M3 - Conference contribution
AN - SCOPUS:85128777909
SN - 9783030995263
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 217
EP - 235
BT - Tools and Algorithms for the Construction and Analysis of Systems - 28th International Conference, TACAS 2022, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022, Proceedings
A2 - Fisman, Dana
A2 - Rosu, Grigore
PB - Springer
T2 - 28th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2022 held as part of 25th European Joint Conferences on Theory and Practice of Software, ETAPS 2022
Y2 - 2 April 2022 through 7 April 2022
ER -