Comparing and combining test-suite reduction and regression test selection

August Shi, Tifany Yung, Alex Gyori, Darko Marinov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Regression testing is widely used to check that changes made to software do not break existing functionality, but regression test suites grow, and running them fully can become costly. Researchers have proposed test-suite reduction and regression test selection as two approaches to reduce this cost by not running some of the tests from the test suite. However, previous research has not empirically evaluated how the two approaches compare to each other, and how well a combination of these approaches performs. We present the first extensive study that compares testsuite reduction and regression test selection approaches individually, and also evaluates a combination of the two approaches. We also propose a new criterion to measure the quality of tests with respect to software changes. Our experiments on 4,793 commits from 17 open-source projects show that regression test selection runs on average fewer tests (by 40.15pp) than test-suite reduction. However, test-suite reduction can have a high loss in fault-detection capability with respect to the changes, whereas a (safe) regression test selection has no loss. The experiments also show that a combination of the two approaches runs even fewer tests (on average 5.34pp) than regression test selection, but these tests still have a loss in fault-detection capability with respect to the changes.

Original languageEnglish (US)
Title of host publication2015 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2015 - Proceedings
PublisherAssociation for Computing Machinery, Inc
Pages237-247
Number of pages11
ISBN (Electronic)9781450336758
DOIs
StatePublished - Aug 30 2015
Event10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2015 - Bergamo, Italy
Duration: Aug 30 2015Sep 4 2015

Publication series

Name2015 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2015 - Proceedings

Other

Other10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE 2015
CountryItaly
CityBergamo
Period8/30/159/4/15

Keywords

  • Regression test selection
  • Regression testing
  • Test-suite reduction

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Comparing and combining test-suite reduction and regression test selection'. Together they form a unique fingerprint.

Cite this