Mutation testing meets approximate computing

Milos Gligoric, Sarfraz Khurshid, Sasa Misailovic, August Shi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

One of the most widely studied techniques in software testing researchis mutation testing - a technique for evaluating the quality of testsuites. Despite over four decades of academic advances in thistechnique, mutation testing has not found its way to mainstreamdevelopment. The key issue with mutation testing is its highcomputational cost: It requires running the test suite against notjust the program under test but against typically thousands ofmutants, i.e., syntactic variants, of the program. Our key insight isthat exciting advances in the upcoming, yet unrelated, area ofapproximate computing allow us to define a principled approach thatprovides the benefits of traditional mutation testing at a fraction ofits usually large cost. This paper introduces the idea of a novel approach, named ApproxiMut, that blends the power of mutation testing with the practicality ofapproximate computing. To demonstrate the potential of our approach, we present a concrete instantiation: Rather than executing testsagainst each mutant on the exact program version, ApproxiMut obtainsan approximate test/program version by applying approximatetransformations and runs tests against each mutant on the approximatedversion. Our initial goal is to (1) measure the correlation betweenmutation scores on the exact and approximate program versions, (2)evaluate the relation among mutation operators and approximatetransformations, (3) discover the best way to approximate a test and aprogram, and (4) evaluate the benefits of ApproxiMut. Our preliminaryresults show similar mutation scores on the exact and approximateprogram versions and uncovered a case when an approximated test was, to our surprise, better than the exact test.

Original languageEnglish (US)
Title of host publicationProceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering
Subtitle of host publicationNew Ideas and Emerging Results Track, ICSE-NIER 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3-6
Number of pages4
ISBN (Electronic)9781538626757
DOIs
StatePublished - Jun 30 2017
Event39th IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017 - Buenos Aires, Argentina
Duration: May 20 2017May 28 2017

Publication series

NameProceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017

Other

Other39th IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017
CountryArgentina
CityBuenos Aires
Period5/20/175/28/17

Fingerprint

Testing
Software testing
Syntactics
Costs
Mutation

Keywords

  • approximate computing
  • approximate transformations
  • mutation testing

ASJC Scopus subject areas

  • Organizational Behavior and Human Resource Management
  • Software
  • Management of Technology and Innovation

Cite this

Gligoric, M., Khurshid, S., Misailovic, S., & Shi, A. (2017). Mutation testing meets approximate computing. In Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017 (pp. 3-6). [7966867] (Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICSE-NIER.2017.15

Mutation testing meets approximate computing. / Gligoric, Milos; Khurshid, Sarfraz; Misailovic, Sasa; Shi, August.

Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 3-6 7966867 (Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Gligoric, M, Khurshid, S, Misailovic, S & Shi, A 2017, Mutation testing meets approximate computing. in Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017., 7966867, Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017, Institute of Electrical and Electronics Engineers Inc., pp. 3-6, 39th IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017, Buenos Aires, Argentina, 5/20/17. https://doi.org/10.1109/ICSE-NIER.2017.15
Gligoric M, Khurshid S, Misailovic S, Shi A. Mutation testing meets approximate computing. In Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 3-6. 7966867. (Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017). https://doi.org/10.1109/ICSE-NIER.2017.15
Gligoric, Milos ; Khurshid, Sarfraz ; Misailovic, Sasa ; Shi, August. / Mutation testing meets approximate computing. Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 3-6 (Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017).
@inproceedings{142c7cfb1be64396a5e8d635ac6fa547,
title = "Mutation testing meets approximate computing",
abstract = "One of the most widely studied techniques in software testing researchis mutation testing - a technique for evaluating the quality of testsuites. Despite over four decades of academic advances in thistechnique, mutation testing has not found its way to mainstreamdevelopment. The key issue with mutation testing is its highcomputational cost: It requires running the test suite against notjust the program under test but against typically thousands ofmutants, i.e., syntactic variants, of the program. Our key insight isthat exciting advances in the upcoming, yet unrelated, area ofapproximate computing allow us to define a principled approach thatprovides the benefits of traditional mutation testing at a fraction ofits usually large cost. This paper introduces the idea of a novel approach, named ApproxiMut, that blends the power of mutation testing with the practicality ofapproximate computing. To demonstrate the potential of our approach, we present a concrete instantiation: Rather than executing testsagainst each mutant on the exact program version, ApproxiMut obtainsan approximate test/program version by applying approximatetransformations and runs tests against each mutant on the approximatedversion. Our initial goal is to (1) measure the correlation betweenmutation scores on the exact and approximate program versions, (2)evaluate the relation among mutation operators and approximatetransformations, (3) discover the best way to approximate a test and aprogram, and (4) evaluate the benefits of ApproxiMut. Our preliminaryresults show similar mutation scores on the exact and approximateprogram versions and uncovered a case when an approximated test was, to our surprise, better than the exact test.",
keywords = "approximate computing, approximate transformations, mutation testing",
author = "Milos Gligoric and Sarfraz Khurshid and Sasa Misailovic and August Shi",
year = "2017",
month = "6",
day = "30",
doi = "10.1109/ICSE-NIER.2017.15",
language = "English (US)",
series = "Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "3--6",
booktitle = "Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering",
address = "United States",

}

TY - GEN

T1 - Mutation testing meets approximate computing

AU - Gligoric, Milos

AU - Khurshid, Sarfraz

AU - Misailovic, Sasa

AU - Shi, August

PY - 2017/6/30

Y1 - 2017/6/30

N2 - One of the most widely studied techniques in software testing researchis mutation testing - a technique for evaluating the quality of testsuites. Despite over four decades of academic advances in thistechnique, mutation testing has not found its way to mainstreamdevelopment. The key issue with mutation testing is its highcomputational cost: It requires running the test suite against notjust the program under test but against typically thousands ofmutants, i.e., syntactic variants, of the program. Our key insight isthat exciting advances in the upcoming, yet unrelated, area ofapproximate computing allow us to define a principled approach thatprovides the benefits of traditional mutation testing at a fraction ofits usually large cost. This paper introduces the idea of a novel approach, named ApproxiMut, that blends the power of mutation testing with the practicality ofapproximate computing. To demonstrate the potential of our approach, we present a concrete instantiation: Rather than executing testsagainst each mutant on the exact program version, ApproxiMut obtainsan approximate test/program version by applying approximatetransformations and runs tests against each mutant on the approximatedversion. Our initial goal is to (1) measure the correlation betweenmutation scores on the exact and approximate program versions, (2)evaluate the relation among mutation operators and approximatetransformations, (3) discover the best way to approximate a test and aprogram, and (4) evaluate the benefits of ApproxiMut. Our preliminaryresults show similar mutation scores on the exact and approximateprogram versions and uncovered a case when an approximated test was, to our surprise, better than the exact test.

AB - One of the most widely studied techniques in software testing researchis mutation testing - a technique for evaluating the quality of testsuites. Despite over four decades of academic advances in thistechnique, mutation testing has not found its way to mainstreamdevelopment. The key issue with mutation testing is its highcomputational cost: It requires running the test suite against notjust the program under test but against typically thousands ofmutants, i.e., syntactic variants, of the program. Our key insight isthat exciting advances in the upcoming, yet unrelated, area ofapproximate computing allow us to define a principled approach thatprovides the benefits of traditional mutation testing at a fraction ofits usually large cost. This paper introduces the idea of a novel approach, named ApproxiMut, that blends the power of mutation testing with the practicality ofapproximate computing. To demonstrate the potential of our approach, we present a concrete instantiation: Rather than executing testsagainst each mutant on the exact program version, ApproxiMut obtainsan approximate test/program version by applying approximatetransformations and runs tests against each mutant on the approximatedversion. Our initial goal is to (1) measure the correlation betweenmutation scores on the exact and approximate program versions, (2)evaluate the relation among mutation operators and approximatetransformations, (3) discover the best way to approximate a test and aprogram, and (4) evaluate the benefits of ApproxiMut. Our preliminaryresults show similar mutation scores on the exact and approximateprogram versions and uncovered a case when an approximated test was, to our surprise, better than the exact test.

KW - approximate computing

KW - approximate transformations

KW - mutation testing

UR - http://www.scopus.com/inward/record.url?scp=85026744114&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85026744114&partnerID=8YFLogxK

U2 - 10.1109/ICSE-NIER.2017.15

DO - 10.1109/ICSE-NIER.2017.15

M3 - Conference contribution

AN - SCOPUS:85026744114

T3 - Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Results Track, ICSE-NIER 2017

SP - 3

EP - 6

BT - Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering

PB - Institute of Electrical and Electronics Engineers Inc.

ER -