Understanding and Improving Regression Test Selection in Continuous Integration

August Shi, Peiyuan Zhao, Darko Marinov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Developers rely on regression testing in their continuous integration (CI) environment to find changes that introduce regression faults. While regression testing is widely practiced, it can be costly. Regression test selection (RTS) reduces the cost of regression testing by not running the tests that are unaffected by the changes. Industry has adopted module-level RTS for their CI environment, while researchers have proposed class-level RTS. In this paper, we compare module-and class-level RTS techniques in a cloud-based CI environment, Travis. We also develop and evaluate a hybrid RTS technique that combines aspects of the module-and class-level RTS techniques. We evaluate all the techniques on real Travis builds. We find that the RTS techniques do save testing time compared to running all tests (RetestAll), but the percentage of time for a full build using RTS (76.0%) is not as low as found in previous work, due to the extra overhead in a cloud-based CI environment. Moreover, we inspect test failures from RetestAll builds, and although we find that RTS techniques can miss to select failed tests, these test failures are almost all flaky test failures. As such, RTS techniques provide additional value in helping developers avoid wasting time debugging failures not related to the recent code changes. Overall, our results show that RTS can be beneficial for the developers in the CI environment, and RTS not only saves time but also avoids misleading developers by flaky test failures.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE 30th International Symposium on Software Reliability Engineering, ISSRE 2019
EditorsKatinka Wolter, Ina Schieferdecker, Barbara Gallina, Michel Cukier, Roberto Natella, Naghmeh Ivaki, Nuno Laranjeiro
PublisherIEEE Computer Society
Pages228-238
Number of pages11
ISBN (Electronic)9781728149813
DOIs
StatePublished - Oct 2019
Event30th IEEE International Symposium on Software Reliability Engineering, ISSRE 2019 - Berlin, Germany
Duration: Oct 28 2019Oct 31 2019

Publication series

NameProceedings - International Symposium on Software Reliability Engineering, ISSRE
Volume2019-October
ISSN (Print)1071-9458

Conference

Conference30th IEEE International Symposium on Software Reliability Engineering, ISSRE 2019
Country/TerritoryGermany
CityBerlin
Period10/28/1910/31/19

Keywords

  • Continuous integration
  • Flaky tests
  • Regression test selection

ASJC Scopus subject areas

  • Software
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'Understanding and Improving Regression Test Selection in Continuous Integration'. Together they form a unique fingerprint.

Cite this