TY - GEN
T1 - Techniques for evolution-aware runtime verification
AU - Legunsen, Owolabi
AU - Zhang, Yi
AU - Hadzi-Tanovic, Milica
AU - Rosu, Grigore
AU - Marinov, Darko
N1 - We thank Milos Gligoric, Alex Gyori, Farah Hariri, Sasa Misailovic, August Shi, Andrei Stefanescu, Tianyin Xu, members of the Formal Systems Laboratory and CS591SE participants at UIUC for discussions. This work was partially supported by National Science Foundation grants CCF-1421503, CCF-1421575, CNS-1619275, CNS-1646305, CNS-1740916, and CCF-1763788. We gratefully acknowledge support from IOHK, Microsoft, Qualcomm, and Runtime Verification,Inc.
PY - 2019/4
Y1 - 2019/4
N2 - Runtime Verification (RV) can help find bugs by monitoring program executions against formal properties. Developers should ideally use RV whenever they run tests, to find more bugs earlier. Despite tremendous research progress, RV still incurs high overhead in (1) machine time to monitor properties and (2) developer time to wait for and inspect violations from test executions that do not satisfy the properties. Moreover, all prior RV techniques consider only one program version and wastefully re-monitor unaffected properties and code as software evolves. We present the first evolution-aware RV techniques that reduce RV overhead across multiple program versions. Regression Property Selection (RPS) re-monitors only properties that can be violated in parts of code affected by changes, reducing machine time and developer time. Violation Message Suppression (VMS) simply shows only new violations to reduce developer time; it does not reduce machine time. Regression Property Prioritization (RPP) splits RV in two phases: properties more likely to find bugs are monitored in a critical phase to provide faster feedback to the developers; the rest are monitored in a background phase. We compare our techniques with the evolution-unaware (base) RV when monitoring test executions in 200 versions of 10 open-source projects. RPS and the RPP critical phase reduce the average RV overhead from 9.4x (for base RV) to 1.8x, without missing any new violations. VMS reduces the average number of violations 540x, from 54 violations per version (for base RV) to one violation per 10 versions.
AB - Runtime Verification (RV) can help find bugs by monitoring program executions against formal properties. Developers should ideally use RV whenever they run tests, to find more bugs earlier. Despite tremendous research progress, RV still incurs high overhead in (1) machine time to monitor properties and (2) developer time to wait for and inspect violations from test executions that do not satisfy the properties. Moreover, all prior RV techniques consider only one program version and wastefully re-monitor unaffected properties and code as software evolves. We present the first evolution-aware RV techniques that reduce RV overhead across multiple program versions. Regression Property Selection (RPS) re-monitors only properties that can be violated in parts of code affected by changes, reducing machine time and developer time. Violation Message Suppression (VMS) simply shows only new violations to reduce developer time; it does not reduce machine time. Regression Property Prioritization (RPP) splits RV in two phases: properties more likely to find bugs are monitored in a critical phase to provide faster feedback to the developers; the rest are monitored in a background phase. We compare our techniques with the evolution-unaware (base) RV when monitoring test executions in 200 versions of 10 open-source projects. RPS and the RPP critical phase reduce the average RV overhead from 9.4x (for base RV) to 1.8x, without missing any new violations. VMS reduces the average number of violations 540x, from 54 violations per version (for base RV) to one violation per 10 versions.
KW - Regression testing
KW - Runtime verification
KW - Software evolution
KW - Software testing
KW - Specifications
UR - http://www.scopus.com/inward/record.url?scp=85067961352&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85067961352&partnerID=8YFLogxK
U2 - 10.1109/ICST.2019.00037
DO - 10.1109/ICST.2019.00037
M3 - Conference contribution
AN - SCOPUS:85067961352
T3 - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
SP - 300
EP - 311
BT - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019
Y2 - 22 April 2019 through 27 April 2019
ER -