TY - GEN
T1 - Detecting Assumptions on Deterministic Implementations of Non-deterministic Specifications
AU - Shi, August
AU - Gyori, Alex
AU - Legunsen, Owolabi
AU - Marinov, Darko
N1 - Publisher Copyright:
© 2016 IEEE.
PY - 2016/7/18
Y1 - 2016/7/18
N2 - Some commonly used methods have nondeterministicspecifications, e.g., iterating through a set canreturn the elements in any order. However, non-deterministicspecifications typically have deterministic implementations, e.g.,iterating through two sets constructed in the same way mayreturn their elements in the same order. We use the termADINS code to refer to code that Assumes a DeterministicImplementation of a method with a Non-deterministic Specification. Such ADINS code can behave unexpectedly whenthe implementation changes, even if the specification remainsthe same. Further, ADINS code can lead to flaky tests-teststhat pass or fail seemingly non-deterministically. We present a simple technique, called NONDEX, for detectingflaky tests due to ADINS code. We implemented NONDEX forJava: we found 31 methods with non-deterministic specificationsin the Java Standard Library, manually built non-deterministicmodels for these methods, and used a modified Java VirtualMachine to explore various non-deterministic choices. We evaluatedNONDEX on 195 open-source projects from GitHub and 72student submissions from a programming homework assignment.NONDEX detected 60 flaky tests in 21 open-source projects and110 flaky tests in 34 student submissions.
AB - Some commonly used methods have nondeterministicspecifications, e.g., iterating through a set canreturn the elements in any order. However, non-deterministicspecifications typically have deterministic implementations, e.g.,iterating through two sets constructed in the same way mayreturn their elements in the same order. We use the termADINS code to refer to code that Assumes a DeterministicImplementation of a method with a Non-deterministic Specification. Such ADINS code can behave unexpectedly whenthe implementation changes, even if the specification remainsthe same. Further, ADINS code can lead to flaky tests-teststhat pass or fail seemingly non-deterministically. We present a simple technique, called NONDEX, for detectingflaky tests due to ADINS code. We implemented NONDEX forJava: we found 31 methods with non-deterministic specificationsin the Java Standard Library, manually built non-deterministicmodels for these methods, and used a modified Java VirtualMachine to explore various non-deterministic choices. We evaluatedNONDEX on 195 open-source projects from GitHub and 72student submissions from a programming homework assignment.NONDEX detected 60 flaky tests in 21 open-source projects and110 flaky tests in 34 student submissions.
KW - Flaky Tests
KW - Regression Testing
KW - Specification
UR - http://www.scopus.com/inward/record.url?scp=84983246845&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84983246845&partnerID=8YFLogxK
U2 - 10.1109/ICST.2016.40
DO - 10.1109/ICST.2016.40
M3 - Conference contribution
AN - SCOPUS:84983246845
T3 - Proceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016
SP - 80
EP - 90
BT - Proceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 9th IEEE International Conference on Software Testing, Verification and Validation, ICST 2016
Y2 - 10 April 2016 through 15 April 2016
ER -