A large-scale study of test coverage evolution

Michael Hilton, Jonathan Bell, Darko Marinov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Statement coverage is commonly used as a measure of test suite quality. Coverage is often used as a part of a code review process: if a patch decreases overall coverage, or is itself not covered, then the patch is scrutinized more closely. Traditional studies of how coverage changes with code evolution have examined the overall coverage of the entire program, and more recent work directly examines the coverage of patches (changed statements). We present an evaluation much larger than prior studies and moreover consider a new, important kind of change - coverage changes of unchanged statements. We present a large-scale evaluation of code coverage evolution over 7,816 builds of 47 projects written in popular languages including Java, Python, and Scala. We find that in large, mature projects, simply measuring the change to statement coverage does not capture the nuances of code evolution. Going beyond considering statement coverage as a simple ratio, we examine how the set of statements covered evolves between project revisions. We present and study new ways to assess the impact of a patch on a project's test suite quality that both separates coverage of the patch from coverage of the non-patch, and separates changes in coverage from changes in the set of statements covered.

Original languageEnglish (US)
Title of host publicationASE 2018 - Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering
EditorsChristian Kastner, Marianne Huchard, Gordon Fraser
PublisherAssociation for Computing Machinery, Inc
Pages53-63
Number of pages11
ISBN (Electronic)9781450359375
DOIs
StatePublished - Sep 3 2018
Event33rd IEEE/ACM International Conference on Automated Software Engineering, ASE 2018 - Montpellier, France
Duration: Sep 3 2018Sep 7 2018

Publication series

NameASE 2018 - Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

Other

Other33rd IEEE/ACM International Conference on Automated Software Engineering, ASE 2018
CountryFrance
CityMontpellier
Period9/3/189/7/18

Keywords

  • Code coverage
  • Empirical study
  • Flaky tests
  • Software testing

ASJC Scopus subject areas

  • Computational Theory and Mathematics
  • Human-Computer Interaction
  • Software

Cite this