TY - GEN
T1 - Balancing trade-offs in test-suite reduction
AU - Shi, August
AU - Gyori, Alex
AU - Gligoric, Milos
AU - Zaytsev, Andrey
AU - Marinov, Darko
N1 - Publisher Copyright:
Copyright 2014 ACM.
PY - 2014/11/16
Y1 - 2014/11/16
N2 - Regression testing is an important activity but can get expensive for large test suites. Test-suite reduction speeds up regression testing by identifying and removing redundant tests based on a given set of requirements. Traditional research on test-suite reduction is rather diverse but most commonly shares three properties: (1) requirements are defined by a coverage criterion such as statement coverage; (2) the reduced test suite has to satisfy all the requirements as the original test suite; and (3) the quality of the reduced test suites is measured on the software version on which the reduction is performed. These properties make it hard for test engineers to decide how to use reduced test suites. We address all three properties of traditional test-suite reduction: (1) we evaluate test-suite reduction with requirements defined by killed mutants; (2) we evaluate inadequate reduction that does not require reduced test suites to satisfy all the requirements; and (3) we propose evolution-aware metrics that evaluate the quality of the reduced test suites across multiple software versions. Our evaluations allow a more thorough exploration of trade-offs in test-suite reduction, and our evolution-aware metrics show how the quality of reduced test suites can change after the version where the reduction is performed. We compare the trade-offs among various reductions on 18 projects with a total of 261,235 tests over 3,590 commits and a cumulative history spanning 35 years of development. Our results help test engineers make a more informed decision about balancing size, coverage, and fault-detection loss of reduced test suites.
AB - Regression testing is an important activity but can get expensive for large test suites. Test-suite reduction speeds up regression testing by identifying and removing redundant tests based on a given set of requirements. Traditional research on test-suite reduction is rather diverse but most commonly shares three properties: (1) requirements are defined by a coverage criterion such as statement coverage; (2) the reduced test suite has to satisfy all the requirements as the original test suite; and (3) the quality of the reduced test suites is measured on the software version on which the reduction is performed. These properties make it hard for test engineers to decide how to use reduced test suites. We address all three properties of traditional test-suite reduction: (1) we evaluate test-suite reduction with requirements defined by killed mutants; (2) we evaluate inadequate reduction that does not require reduced test suites to satisfy all the requirements; and (3) we propose evolution-aware metrics that evaluate the quality of the reduced test suites across multiple software versions. Our evaluations allow a more thorough exploration of trade-offs in test-suite reduction, and our evolution-aware metrics show how the quality of reduced test suites can change after the version where the reduction is performed. We compare the trade-offs among various reductions on 18 projects with a total of 261,235 tests over 3,590 commits and a cumulative history spanning 35 years of development. Our results help test engineers make a more informed decision about balancing size, coverage, and fault-detection loss of reduced test suites.
KW - Software evolution
KW - Test-suite reduction
UR - http://www.scopus.com/inward/record.url?scp=84986890458&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84986890458&partnerID=8YFLogxK
U2 - 10.1145/2635868.2635921
DO - 10.1145/2635868.2635921
M3 - Conference contribution
AN - SCOPUS:84986890458
T3 - Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering
SP - 246
EP - 256
BT - 22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering, FSE 2014 - Proceedings
PB - Association for Computing Machinery
T2 - 22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering, FSE 2014
Y2 - 16 November 2014 through 21 November 2014
ER -