Balancing trade-offs in test-suite reduction

August Shi, Alex Gyori, Milos Gligoric, Andrey Zaytsev, Darko Marinov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Regression testing is an important activity but can get expensive for large test suites. Test-suite reduction speeds up regression testing by identifying and removing redundant tests based on a given set of requirements. Traditional research on test-suite reduction is rather diverse but most commonly shares three properties: (1) requirements are defined by a coverage criterion such as statement coverage; (2) the reduced test suite has to satisfy all the requirements as the original test suite; and (3) the quality of the reduced test suites is measured on the software version on which the reduction is performed. These properties make it hard for test engineers to decide how to use reduced test suites. We address all three properties of traditional test-suite reduction: (1) we evaluate test-suite reduction with requirements defined by killed mutants; (2) we evaluate inadequate reduction that does not require reduced test suites to satisfy all the requirements; and (3) we propose evolution-aware metrics that evaluate the quality of the reduced test suites across multiple software versions. Our evaluations allow a more thorough exploration of trade-offs in test-suite reduction, and our evolution-aware metrics show how the quality of reduced test suites can change after the version where the reduction is performed. We compare the trade-offs among various reductions on 18 projects with a total of 261,235 tests over 3,590 commits and a cumulative history spanning 35 years of development. Our results help test engineers make a more informed decision about balancing size, coverage, and fault-detection loss of reduced test suites.

Original languageEnglish (US)
Title of host publication22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering, FSE 2014 - Proceedings
PublisherAssociation for Computing Machinery
Pages246-256
Number of pages11
ISBN (Electronic)9781450330565
DOIs
StatePublished - Nov 16 2014
Event22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering, FSE 2014 - Hong Kong, China
Duration: Nov 16 2014Nov 21 2014

Publication series

NameProceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering
Volume16-21-November-2014

Other

Other22nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering, FSE 2014
CountryChina
CityHong Kong
Period11/16/1411/21/14

Keywords

  • Software evolution
  • Test-suite reduction

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Balancing trade-offs in test-suite reduction'. Together they form a unique fingerprint.

Cite this