TY - GEN
T1 - Comparing mutation testing at the levels of source code and compiler intermediate representation
AU - Hariri, Farah
AU - Shi, August
AU - Fernando, Vimuth
AU - Mahmood, Suleman
AU - Marinov, Darko
N1 - Funding Information:
ACKNOWLEDGEMENTS We thank Sarfraz Khurshid, Sasa Misailovic, and Tao Xie for their discussions about this project. This research was partially supported by NSF Grant Nos. CCF-1421503, CCF-1629431, CNS-1646305, and CNS-1740916.
Publisher Copyright:
© 2019 IEEE.
PY - 2019/4
Y1 - 2019/4
N2 - Mutation testing is widely used in research for evaluating the effectiveness of test suites. There are multiple mutation tools that perform mutation at different levels, including traditional mutation testing at the level of source code (SRC) and more recent mutation testing at the level of compiler intermediate representation (IR). This paper presents an extensive comparison of mutation testing at the SRC and IR levels, specifically at the C programming language and the LLVM compiler IR levels. We use a mutation testing tool called SRCIROR that implements conceptually the same mutation operators at both levels. We also employ automated techniques to account for equivalent and duplicated mutants, and to determine minimal and surface mutants. We carry out our study on 15 programs from the Coreutils library. Overall, we find mutation testing to be better at the SRC level: the SRC level produces much fewer mutants and is thus less expensive, but the SRC level still generates a similar number of minimal and surface mutants, and the mutation scores at both levels are very closely correlated. We also perform a case study on the Space program to evaluate which level's mutation score correlates better with the actual fault-detection capability of test suites sampled from Space's test pool. We find the mutation score at both levels to not be very correlated with the actual fault-detection capability of test suites.
AB - Mutation testing is widely used in research for evaluating the effectiveness of test suites. There are multiple mutation tools that perform mutation at different levels, including traditional mutation testing at the level of source code (SRC) and more recent mutation testing at the level of compiler intermediate representation (IR). This paper presents an extensive comparison of mutation testing at the SRC and IR levels, specifically at the C programming language and the LLVM compiler IR levels. We use a mutation testing tool called SRCIROR that implements conceptually the same mutation operators at both levels. We also employ automated techniques to account for equivalent and duplicated mutants, and to determine minimal and surface mutants. We carry out our study on 15 programs from the Coreutils library. Overall, we find mutation testing to be better at the SRC level: the SRC level produces much fewer mutants and is thus less expensive, but the SRC level still generates a similar number of minimal and surface mutants, and the mutation scores at both levels are very closely correlated. We also perform a case study on the Space program to evaluate which level's mutation score correlates better with the actual fault-detection capability of test suites sampled from Space's test pool. We find the mutation score at both levels to not be very correlated with the actual fault-detection capability of test suites.
KW - Intermediate representation
KW - LLVM
KW - Mutation testing
KW - SRCIROR
UR - http://www.scopus.com/inward/record.url?scp=85067933235&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85067933235&partnerID=8YFLogxK
U2 - 10.1109/ICST.2019.00021
DO - 10.1109/ICST.2019.00021
M3 - Conference contribution
AN - SCOPUS:85067933235
T3 - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
SP - 114
EP - 124
BT - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019
Y2 - 22 April 2019 through 27 April 2019
ER -