TY - GEN
T1 - How does regression test prioritization perform in real-world software evolution?
AU - Lu, Yafeng
AU - Lou, Yiling
AU - Cheng, Shiyang
AU - Zhang, Lingming
AU - Hao, Dan
AU - Zhou, Yangfan
AU - Zhang, Lu
N1 - Publisher Copyright:
© 2016 ACM.
PY - 2016/5/14
Y1 - 2016/5/14
N2 - In recent years, researchers have intensively investigated various topics in test prioritization, which aims to re-order tests to increase the rate of fault detection during regression testing. While the main research focus in test prioritization is on proposing novel prioritization techniques and evaluating on more and larger subject systems, little e.ort has been put on investigating the threats to validity in existing work on test prioritization. One main threat to validity is that existing work mainly evaluates prioritization techniques based on simple artificial changes on the source code and tests. For example, the changes in the source code usually include only seeded program faults, whereas the test suite is usually not augmented at all. On the contrary, in real-world software development, software systems usually undergo various changes on the source code and test suite augmentation. Therefore, it is not clear whether the conclusions drawn by existing work in test prioritization from the artificial changes are still valid for real-world software evolution. In this paper, we present the first empirical study to investigate this important threat to validity in test prioritization. We reimplemented 24 variant techniques of both the traditional and time-aware test prioritization, and investigated the impacts of software evolution on those techniques based on the version history of 8 real-world Java programs from GitHub. The results show that for both traditional and time-aware test prioritization, test suite augmentation significantly hampers their e.ectiveness, whereas source code changes alone do not influence their e.ectiveness much.
AB - In recent years, researchers have intensively investigated various topics in test prioritization, which aims to re-order tests to increase the rate of fault detection during regression testing. While the main research focus in test prioritization is on proposing novel prioritization techniques and evaluating on more and larger subject systems, little e.ort has been put on investigating the threats to validity in existing work on test prioritization. One main threat to validity is that existing work mainly evaluates prioritization techniques based on simple artificial changes on the source code and tests. For example, the changes in the source code usually include only seeded program faults, whereas the test suite is usually not augmented at all. On the contrary, in real-world software development, software systems usually undergo various changes on the source code and test suite augmentation. Therefore, it is not clear whether the conclusions drawn by existing work in test prioritization from the artificial changes are still valid for real-world software evolution. In this paper, we present the first empirical study to investigate this important threat to validity in test prioritization. We reimplemented 24 variant techniques of both the traditional and time-aware test prioritization, and investigated the impacts of software evolution on those techniques based on the version history of 8 real-world Java programs from GitHub. The results show that for both traditional and time-aware test prioritization, test suite augmentation significantly hampers their e.ectiveness, whereas source code changes alone do not influence their e.ectiveness much.
UR - http://www.scopus.com/inward/record.url?scp=84971393429&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84971393429&partnerID=8YFLogxK
U2 - 10.1145/2884781.2884874
DO - 10.1145/2884781.2884874
M3 - Conference contribution
AN - SCOPUS:84971393429
T3 - Proceedings - International Conference on Software Engineering
SP - 535
EP - 546
BT - Proceedings - 2016 IEEE/ACM 38th IEEE International Conference on Software Engineering Companion, ICSE 2016
PB - IEEE Computer Society
T2 - 2016 IEEE/ACM 38th IEEE International Conference on Software Engineering, ICSE 2016
Y2 - 14 May 2016 through 22 May 2016
ER -