Resurgence of Regression Test Selection for C++

Ben Fu, Sasa Misailovic, Milos Gligoric

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Regression testing - running available tests after each project change - is widely practiced in industry. Despite its widespread use and importance, regression testing is a costly activity. Regression test selection (RTS) optimizes regression testing by selecting only tests affected by project changes. RTS has been extensively studied and several tools have been deployed in large projects. However, work on RTS over the last decade has mostly focused on languages with abstract computing machines (e.g., JVM). Meanwhile development practices (e.g., frequency of commits, testing frameworks, compilers) in C++ projects have dramatically changed and the way we should design and implement RTS tools and the benefits of those tools is unknown. We present a design and implementation of an RTS technique, dubbed RTS++, that targets projects written in C++, which compile to LLVM IR and use the Google Test testing framework. RTS++ uses static analysis of a function call graph to select tests. RTS++ integrates with many existing build systems, including AutoMake, CMake, and Make. We evaluated RTS++ on 11 large open-source projects, totaling 3,811,916 lines of code. To the best of our knowledge, this is the largest evaluation of an RTS technique for C++. We measured the benefits of RTS++ compared to running all available tests (i.e., retest-all). Our results show that RTS++ reduces the number of executed tests and end-to-end testing time by 88% and 61% on average.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages323-334
Number of pages12
ISBN (Electronic)9781728117355
DOIs
StatePublished - Apr 2019
Event12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019 - Xi'an, China
Duration: Apr 22 2019Apr 27 2019

Publication series

NameProceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

Conference

Conference12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019
CountryChina
CityXi'an
Period4/22/194/27/19

Fingerprint

Testing
Static analysis
Industry

Keywords

  • Call graph
  • Google Test
  • LLVM
  • Regression test selection
  • Static analysis

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Safety, Risk, Reliability and Quality

Cite this

Fu, B., Misailovic, S., & Gligoric, M. (2019). Resurgence of Regression Test Selection for C++. In Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019 (pp. 323-334). [8730161] (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICST.2019.00039

Resurgence of Regression Test Selection for C++. / Fu, Ben; Misailovic, Sasa; Gligoric, Milos.

Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 323-334 8730161 (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Fu, B, Misailovic, S & Gligoric, M 2019, Resurgence of Regression Test Selection for C++. in Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019., 8730161, Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019, Institute of Electrical and Electronics Engineers Inc., pp. 323-334, 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019, Xi'an, China, 4/22/19. https://doi.org/10.1109/ICST.2019.00039
Fu B, Misailovic S, Gligoric M. Resurgence of Regression Test Selection for C++. In Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 323-334. 8730161. (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019). https://doi.org/10.1109/ICST.2019.00039
Fu, Ben ; Misailovic, Sasa ; Gligoric, Milos. / Resurgence of Regression Test Selection for C++. Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 323-334 (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019).
@inproceedings{cb21451cbb864f6694ff8de675993171,
title = "Resurgence of Regression Test Selection for C++",
abstract = "Regression testing - running available tests after each project change - is widely practiced in industry. Despite its widespread use and importance, regression testing is a costly activity. Regression test selection (RTS) optimizes regression testing by selecting only tests affected by project changes. RTS has been extensively studied and several tools have been deployed in large projects. However, work on RTS over the last decade has mostly focused on languages with abstract computing machines (e.g., JVM). Meanwhile development practices (e.g., frequency of commits, testing frameworks, compilers) in C++ projects have dramatically changed and the way we should design and implement RTS tools and the benefits of those tools is unknown. We present a design and implementation of an RTS technique, dubbed RTS++, that targets projects written in C++, which compile to LLVM IR and use the Google Test testing framework. RTS++ uses static analysis of a function call graph to select tests. RTS++ integrates with many existing build systems, including AutoMake, CMake, and Make. We evaluated RTS++ on 11 large open-source projects, totaling 3,811,916 lines of code. To the best of our knowledge, this is the largest evaluation of an RTS technique for C++. We measured the benefits of RTS++ compared to running all available tests (i.e., retest-all). Our results show that RTS++ reduces the number of executed tests and end-to-end testing time by 88{\%} and 61{\%} on average.",
keywords = "Call graph, Google Test, LLVM, Regression test selection, Static analysis",
author = "Ben Fu and Sasa Misailovic and Milos Gligoric",
year = "2019",
month = "4",
doi = "10.1109/ICST.2019.00039",
language = "English (US)",
series = "Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "323--334",
booktitle = "Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019",
address = "United States",

}

TY - GEN

T1 - Resurgence of Regression Test Selection for C++

AU - Fu, Ben

AU - Misailovic, Sasa

AU - Gligoric, Milos

PY - 2019/4

Y1 - 2019/4

N2 - Regression testing - running available tests after each project change - is widely practiced in industry. Despite its widespread use and importance, regression testing is a costly activity. Regression test selection (RTS) optimizes regression testing by selecting only tests affected by project changes. RTS has been extensively studied and several tools have been deployed in large projects. However, work on RTS over the last decade has mostly focused on languages with abstract computing machines (e.g., JVM). Meanwhile development practices (e.g., frequency of commits, testing frameworks, compilers) in C++ projects have dramatically changed and the way we should design and implement RTS tools and the benefits of those tools is unknown. We present a design and implementation of an RTS technique, dubbed RTS++, that targets projects written in C++, which compile to LLVM IR and use the Google Test testing framework. RTS++ uses static analysis of a function call graph to select tests. RTS++ integrates with many existing build systems, including AutoMake, CMake, and Make. We evaluated RTS++ on 11 large open-source projects, totaling 3,811,916 lines of code. To the best of our knowledge, this is the largest evaluation of an RTS technique for C++. We measured the benefits of RTS++ compared to running all available tests (i.e., retest-all). Our results show that RTS++ reduces the number of executed tests and end-to-end testing time by 88% and 61% on average.

AB - Regression testing - running available tests after each project change - is widely practiced in industry. Despite its widespread use and importance, regression testing is a costly activity. Regression test selection (RTS) optimizes regression testing by selecting only tests affected by project changes. RTS has been extensively studied and several tools have been deployed in large projects. However, work on RTS over the last decade has mostly focused on languages with abstract computing machines (e.g., JVM). Meanwhile development practices (e.g., frequency of commits, testing frameworks, compilers) in C++ projects have dramatically changed and the way we should design and implement RTS tools and the benefits of those tools is unknown. We present a design and implementation of an RTS technique, dubbed RTS++, that targets projects written in C++, which compile to LLVM IR and use the Google Test testing framework. RTS++ uses static analysis of a function call graph to select tests. RTS++ integrates with many existing build systems, including AutoMake, CMake, and Make. We evaluated RTS++ on 11 large open-source projects, totaling 3,811,916 lines of code. To the best of our knowledge, this is the largest evaluation of an RTS technique for C++. We measured the benefits of RTS++ compared to running all available tests (i.e., retest-all). Our results show that RTS++ reduces the number of executed tests and end-to-end testing time by 88% and 61% on average.

KW - Call graph

KW - Google Test

KW - LLVM

KW - Regression test selection

KW - Static analysis

UR - http://www.scopus.com/inward/record.url?scp=85067994648&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067994648&partnerID=8YFLogxK

U2 - 10.1109/ICST.2019.00039

DO - 10.1109/ICST.2019.00039

M3 - Conference contribution

AN - SCOPUS:85067994648

T3 - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

SP - 323

EP - 334

BT - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -