TY - GEN
T1 - An extensive study of static regression test selection in modern software evolution
AU - Legunsen, Owolabi
AU - Hariri, Farah
AU - Shi, August
AU - Lu, Yafeng
AU - Zhang, Lingming
AU - Marinov, Darko
PY - 2016/11/1
Y1 - 2016/11/1
N2 - Regression test selection (RTS) aims to reduce regression testing time by only re-running the tests affected by code changes. Prior research on RTS can be broadly split into dy-namic and static techniques. A recently developed dynamic RTS technique called Ekstazi is gaining some adoption in practice, and its evaluation shows that selecting tests at a coarser, class-level granularity provides better results than selecting tests at a finer, method-level granularity. As dy-namic RTS is gaining adoption, it is timely to also evaluate static RTS techniques, some of which were proposed over three decades ago but not extensively evaluated on modern software projects. This paper presents the first extensive study that evalu-ates the performance benefits of static RTS techniques and their safety; a technique is safe if it selects to run all tests that may be affected by code changes. We implemented two static RTS techniques, one class-level and one method-level, and compare several variants of these techniques. We also compare these static RTS techniques against Ek-stazi, a state-of-The-Art, class-level, dynamic RTS technique. The experimental results on 985 revisions of 22 open-source projects show that the class-level static RTS technique is comparable to Ekstazi, with similar performance benefits, but at the risk of being unsafe sometimes. In contrast, the method-level static RTS technique performs rather poorly.
AB - Regression test selection (RTS) aims to reduce regression testing time by only re-running the tests affected by code changes. Prior research on RTS can be broadly split into dy-namic and static techniques. A recently developed dynamic RTS technique called Ekstazi is gaining some adoption in practice, and its evaluation shows that selecting tests at a coarser, class-level granularity provides better results than selecting tests at a finer, method-level granularity. As dy-namic RTS is gaining adoption, it is timely to also evaluate static RTS techniques, some of which were proposed over three decades ago but not extensively evaluated on modern software projects. This paper presents the first extensive study that evalu-ates the performance benefits of static RTS techniques and their safety; a technique is safe if it selects to run all tests that may be affected by code changes. We implemented two static RTS techniques, one class-level and one method-level, and compare several variants of these techniques. We also compare these static RTS techniques against Ek-stazi, a state-of-The-Art, class-level, dynamic RTS technique. The experimental results on 985 revisions of 22 open-source projects show that the class-level static RTS technique is comparable to Ekstazi, with similar performance benefits, but at the risk of being unsafe sometimes. In contrast, the method-level static RTS technique performs rather poorly.
KW - Class firewall
KW - Regression test selection
KW - Static analysis
UR - http://www.scopus.com/inward/record.url?scp=84997112179&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84997112179&partnerID=8YFLogxK
U2 - 10.1145/2950290.2950361
DO - 10.1145/2950290.2950361
M3 - Conference contribution
AN - SCOPUS:84997112179
T3 - Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering
SP - 583
EP - 594
BT - FSE 2016 - Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering
A2 - Su, Zhendong
A2 - Zimmermann, Thomas
A2 - Cleland-Huang, Jane
PB - Association for Computing Machinery
T2 - 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016
Y2 - 13 November 2016 through 18 November 2016
ER -