Abstract
Runtime Verification (RV) can help find bugs by monitoring program executions against formal properties. Developers should ideally use RV whenever they run tests, to find more bugs earlier. Despite tremendous research progress, RV still incurs high overhead in (1) machine time to monitor properties and (2) developer time to wait for and inspect violations from test executions that do not satisfy the properties. Moreover, all prior RV techniques consider only one program version and wastefully re-monitor unaffected properties and code as software evolves. We present the first evolution-aware RV techniques that reduce RV overhead across multiple program versions. Regression Property Selection (RPS) re-monitors only properties that can be violated in parts of code affected by changes, reducing machine time and developer time. Violation Message Suppression (VMS) simply shows only new violations to reduce developer time; it does not reduce machine time. Regression Property Prioritization (RPP) splits RV in two phases: properties more likely to find bugs are monitored in a critical phase to provide faster feedback to the developers; the rest are monitored in a background phase. We compare our techniques with the evolution-unaware (base) RV when monitoring test executions in 200 versions of 10 open-source projects. RPS and the RPP critical phase reduce the average RV overhead from 9.4x (for base RV) to 1.8x, without missing any new violations. VMS reduces the average number of violations 540x, from 54 violations per version (for base RV) to one violation per 10 versions.
| Original language | English (US) |
|---|---|
| Title of host publication | Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| Pages | 300-311 |
| Number of pages | 12 |
| ISBN (Electronic) | 9781728117355 |
| DOIs | |
| State | Published - Apr 2019 |
| Event | 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019 - Xi'an, China Duration: Apr 22 2019 → Apr 27 2019 |
Publication series
| Name | Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019 |
|---|
Conference
| Conference | 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019 |
|---|---|
| Country | China |
| City | Xi'an |
| Period | 4/22/19 → 4/27/19 |
Fingerprint
Keywords
- Regression testing
- Runtime verification
- Software evolution
- Software testing
- Specifications
ASJC Scopus subject areas
- Artificial Intelligence
- Software
- Safety, Risk, Reliability and Quality
Cite this
Techniques for evolution-aware runtime verification. / Legunsen, Owolabi; Zhang, Yi; Hadzi-Tanovic, Milica; Rosu, Grigore; Marinov, Darko.
Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 300-311 8730172 (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Techniques for evolution-aware runtime verification
AU - Legunsen, Owolabi
AU - Zhang, Yi
AU - Hadzi-Tanovic, Milica
AU - Rosu, Grigore
AU - Marinov, Darko
PY - 2019/4
Y1 - 2019/4
N2 - Runtime Verification (RV) can help find bugs by monitoring program executions against formal properties. Developers should ideally use RV whenever they run tests, to find more bugs earlier. Despite tremendous research progress, RV still incurs high overhead in (1) machine time to monitor properties and (2) developer time to wait for and inspect violations from test executions that do not satisfy the properties. Moreover, all prior RV techniques consider only one program version and wastefully re-monitor unaffected properties and code as software evolves. We present the first evolution-aware RV techniques that reduce RV overhead across multiple program versions. Regression Property Selection (RPS) re-monitors only properties that can be violated in parts of code affected by changes, reducing machine time and developer time. Violation Message Suppression (VMS) simply shows only new violations to reduce developer time; it does not reduce machine time. Regression Property Prioritization (RPP) splits RV in two phases: properties more likely to find bugs are monitored in a critical phase to provide faster feedback to the developers; the rest are monitored in a background phase. We compare our techniques with the evolution-unaware (base) RV when monitoring test executions in 200 versions of 10 open-source projects. RPS and the RPP critical phase reduce the average RV overhead from 9.4x (for base RV) to 1.8x, without missing any new violations. VMS reduces the average number of violations 540x, from 54 violations per version (for base RV) to one violation per 10 versions.
AB - Runtime Verification (RV) can help find bugs by monitoring program executions against formal properties. Developers should ideally use RV whenever they run tests, to find more bugs earlier. Despite tremendous research progress, RV still incurs high overhead in (1) machine time to monitor properties and (2) developer time to wait for and inspect violations from test executions that do not satisfy the properties. Moreover, all prior RV techniques consider only one program version and wastefully re-monitor unaffected properties and code as software evolves. We present the first evolution-aware RV techniques that reduce RV overhead across multiple program versions. Regression Property Selection (RPS) re-monitors only properties that can be violated in parts of code affected by changes, reducing machine time and developer time. Violation Message Suppression (VMS) simply shows only new violations to reduce developer time; it does not reduce machine time. Regression Property Prioritization (RPP) splits RV in two phases: properties more likely to find bugs are monitored in a critical phase to provide faster feedback to the developers; the rest are monitored in a background phase. We compare our techniques with the evolution-unaware (base) RV when monitoring test executions in 200 versions of 10 open-source projects. RPS and the RPP critical phase reduce the average RV overhead from 9.4x (for base RV) to 1.8x, without missing any new violations. VMS reduces the average number of violations 540x, from 54 violations per version (for base RV) to one violation per 10 versions.
KW - Regression testing
KW - Runtime verification
KW - Software evolution
KW - Software testing
KW - Specifications
UR - http://www.scopus.com/inward/record.url?scp=85067961352&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85067961352&partnerID=8YFLogxK
U2 - 10.1109/ICST.2019.00037
DO - 10.1109/ICST.2019.00037
M3 - Conference contribution
AN - SCOPUS:85067961352
T3 - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
SP - 300
EP - 311
BT - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
PB - Institute of Electrical and Electronics Engineers Inc.
ER -