TY - GEN
T1 - A large-scale study of test coverage evolution
AU - Hilton, Michael
AU - Bell, Jonathan
AU - Marinov, Darko
N1 - Funding Information:
We thank Hayder Al Haddad and Owolabi Legunsen for discussions about this work. Darko Marinov’s group is supported by NSF grants CCF-1409423, CCF-1421503, CNS-1646305, and CNS-1740916; and gifts from Google and Qualcomm.
Funding Information:
We thank Hayder Al Haddad and Owolabi Legunsen for discussions about this work. Darko Marinov's group is supported by NSF grants CCF-1409423, CCF-1421503, CNS-1646305, and CNS-1740916; and gifts from Google and Qualcomm.
Publisher Copyright:
© 2018 Copyright held by the owner/author(s).
PY - 2018/9/3
Y1 - 2018/9/3
N2 - Statement coverage is commonly used as a measure of test suite quality. Coverage is often used as a part of a code review process: if a patch decreases overall coverage, or is itself not covered, then the patch is scrutinized more closely. Traditional studies of how coverage changes with code evolution have examined the overall coverage of the entire program, and more recent work directly examines the coverage of patches (changed statements). We present an evaluation much larger than prior studies and moreover consider a new, important kind of change - coverage changes of unchanged statements. We present a large-scale evaluation of code coverage evolution over 7,816 builds of 47 projects written in popular languages including Java, Python, and Scala. We find that in large, mature projects, simply measuring the change to statement coverage does not capture the nuances of code evolution. Going beyond considering statement coverage as a simple ratio, we examine how the set of statements covered evolves between project revisions. We present and study new ways to assess the impact of a patch on a project's test suite quality that both separates coverage of the patch from coverage of the non-patch, and separates changes in coverage from changes in the set of statements covered.
AB - Statement coverage is commonly used as a measure of test suite quality. Coverage is often used as a part of a code review process: if a patch decreases overall coverage, or is itself not covered, then the patch is scrutinized more closely. Traditional studies of how coverage changes with code evolution have examined the overall coverage of the entire program, and more recent work directly examines the coverage of patches (changed statements). We present an evaluation much larger than prior studies and moreover consider a new, important kind of change - coverage changes of unchanged statements. We present a large-scale evaluation of code coverage evolution over 7,816 builds of 47 projects written in popular languages including Java, Python, and Scala. We find that in large, mature projects, simply measuring the change to statement coverage does not capture the nuances of code evolution. Going beyond considering statement coverage as a simple ratio, we examine how the set of statements covered evolves between project revisions. We present and study new ways to assess the impact of a patch on a project's test suite quality that both separates coverage of the patch from coverage of the non-patch, and separates changes in coverage from changes in the set of statements covered.
KW - Code coverage
KW - Empirical study
KW - Flaky tests
KW - Software testing
UR - http://www.scopus.com/inward/record.url?scp=85056510917&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85056510917&partnerID=8YFLogxK
U2 - 10.1145/3238147.3238183
DO - 10.1145/3238147.3238183
M3 - Conference contribution
AN - SCOPUS:85056510917
T3 - ASE 2018 - Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering
SP - 53
EP - 63
BT - ASE 2018 - Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering
A2 - Kastner, Christian
A2 - Huchard, Marianne
A2 - Fraser, Gordon
PB - Association for Computing Machinery, Inc
T2 - 33rd IEEE/ACM International Conference on Automated Software Engineering, ASE 2018
Y2 - 3 September 2018 through 7 September 2018
ER -