Techniques for evolution-aware runtime verification

Owolabi Legunsen, Yi Zhang, Milica Hadzi-Tanovic, Grigore Rosu, Darko Marinov

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Runtime Verification (RV) can help find bugs by monitoring program executions against formal properties. Developers should ideally use RV whenever they run tests, to find more bugs earlier. Despite tremendous research progress, RV still incurs high overhead in (1) machine time to monitor properties and (2) developer time to wait for and inspect violations from test executions that do not satisfy the properties. Moreover, all prior RV techniques consider only one program version and wastefully re-monitor unaffected properties and code as software evolves. We present the first evolution-aware RV techniques that reduce RV overhead across multiple program versions. Regression Property Selection (RPS) re-monitors only properties that can be violated in parts of code affected by changes, reducing machine time and developer time. Violation Message Suppression (VMS) simply shows only new violations to reduce developer time; it does not reduce machine time. Regression Property Prioritization (RPP) splits RV in two phases: properties more likely to find bugs are monitored in a critical phase to provide faster feedback to the developers; the rest are monitored in a background phase. We compare our techniques with the evolution-unaware (base) RV when monitoring test executions in 200 versions of 10 open-source projects. RPS and the RPP critical phase reduce the average RV overhead from 9.4x (for base RV) to 1.8x, without missing any new violations. VMS reduces the average number of violations 540x, from 54 violations per version (for base RV) to one violation per 10 versions.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages300-311
Number of pages12
ISBN (Electronic)9781728117355
DOIs
StatePublished - Apr 2019
Event12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019 - Xi'an, China
Duration: Apr 22 2019Apr 27 2019

Publication series

NameProceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

Conference

Conference12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019
CountryChina
CityXi'an
Period4/22/194/27/19

Fingerprint

Monitoring
Feedback

Keywords

  • Regression testing
  • Runtime verification
  • Software evolution
  • Software testing
  • Specifications

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Safety, Risk, Reliability and Quality

Cite this

Legunsen, O., Zhang, Y., Hadzi-Tanovic, M., Rosu, G., & Marinov, D. (2019). Techniques for evolution-aware runtime verification. In Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019 (pp. 300-311). [8730172] (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICST.2019.00037

Techniques for evolution-aware runtime verification. / Legunsen, Owolabi; Zhang, Yi; Hadzi-Tanovic, Milica; Rosu, Grigore; Marinov, Darko.

Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 300-311 8730172 (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Legunsen, O, Zhang, Y, Hadzi-Tanovic, M, Rosu, G & Marinov, D 2019, Techniques for evolution-aware runtime verification. in Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019., 8730172, Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019, Institute of Electrical and Electronics Engineers Inc., pp. 300-311, 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019, Xi'an, China, 4/22/19. https://doi.org/10.1109/ICST.2019.00037
Legunsen O, Zhang Y, Hadzi-Tanovic M, Rosu G, Marinov D. Techniques for evolution-aware runtime verification. In Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 300-311. 8730172. (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019). https://doi.org/10.1109/ICST.2019.00037
Legunsen, Owolabi ; Zhang, Yi ; Hadzi-Tanovic, Milica ; Rosu, Grigore ; Marinov, Darko. / Techniques for evolution-aware runtime verification. Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 300-311 (Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019).
@inproceedings{aae8e69065e04e01b253396abe374cb6,
title = "Techniques for evolution-aware runtime verification",
abstract = "Runtime Verification (RV) can help find bugs by monitoring program executions against formal properties. Developers should ideally use RV whenever they run tests, to find more bugs earlier. Despite tremendous research progress, RV still incurs high overhead in (1) machine time to monitor properties and (2) developer time to wait for and inspect violations from test executions that do not satisfy the properties. Moreover, all prior RV techniques consider only one program version and wastefully re-monitor unaffected properties and code as software evolves. We present the first evolution-aware RV techniques that reduce RV overhead across multiple program versions. Regression Property Selection (RPS) re-monitors only properties that can be violated in parts of code affected by changes, reducing machine time and developer time. Violation Message Suppression (VMS) simply shows only new violations to reduce developer time; it does not reduce machine time. Regression Property Prioritization (RPP) splits RV in two phases: properties more likely to find bugs are monitored in a critical phase to provide faster feedback to the developers; the rest are monitored in a background phase. We compare our techniques with the evolution-unaware (base) RV when monitoring test executions in 200 versions of 10 open-source projects. RPS and the RPP critical phase reduce the average RV overhead from 9.4x (for base RV) to 1.8x, without missing any new violations. VMS reduces the average number of violations 540x, from 54 violations per version (for base RV) to one violation per 10 versions.",
keywords = "Regression testing, Runtime verification, Software evolution, Software testing, Specifications",
author = "Owolabi Legunsen and Yi Zhang and Milica Hadzi-Tanovic and Grigore Rosu and Darko Marinov",
year = "2019",
month = "4",
doi = "10.1109/ICST.2019.00037",
language = "English (US)",
series = "Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "300--311",
booktitle = "Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019",
address = "United States",

}

TY - GEN

T1 - Techniques for evolution-aware runtime verification

AU - Legunsen, Owolabi

AU - Zhang, Yi

AU - Hadzi-Tanovic, Milica

AU - Rosu, Grigore

AU - Marinov, Darko

PY - 2019/4

Y1 - 2019/4

N2 - Runtime Verification (RV) can help find bugs by monitoring program executions against formal properties. Developers should ideally use RV whenever they run tests, to find more bugs earlier. Despite tremendous research progress, RV still incurs high overhead in (1) machine time to monitor properties and (2) developer time to wait for and inspect violations from test executions that do not satisfy the properties. Moreover, all prior RV techniques consider only one program version and wastefully re-monitor unaffected properties and code as software evolves. We present the first evolution-aware RV techniques that reduce RV overhead across multiple program versions. Regression Property Selection (RPS) re-monitors only properties that can be violated in parts of code affected by changes, reducing machine time and developer time. Violation Message Suppression (VMS) simply shows only new violations to reduce developer time; it does not reduce machine time. Regression Property Prioritization (RPP) splits RV in two phases: properties more likely to find bugs are monitored in a critical phase to provide faster feedback to the developers; the rest are monitored in a background phase. We compare our techniques with the evolution-unaware (base) RV when monitoring test executions in 200 versions of 10 open-source projects. RPS and the RPP critical phase reduce the average RV overhead from 9.4x (for base RV) to 1.8x, without missing any new violations. VMS reduces the average number of violations 540x, from 54 violations per version (for base RV) to one violation per 10 versions.

AB - Runtime Verification (RV) can help find bugs by monitoring program executions against formal properties. Developers should ideally use RV whenever they run tests, to find more bugs earlier. Despite tremendous research progress, RV still incurs high overhead in (1) machine time to monitor properties and (2) developer time to wait for and inspect violations from test executions that do not satisfy the properties. Moreover, all prior RV techniques consider only one program version and wastefully re-monitor unaffected properties and code as software evolves. We present the first evolution-aware RV techniques that reduce RV overhead across multiple program versions. Regression Property Selection (RPS) re-monitors only properties that can be violated in parts of code affected by changes, reducing machine time and developer time. Violation Message Suppression (VMS) simply shows only new violations to reduce developer time; it does not reduce machine time. Regression Property Prioritization (RPP) splits RV in two phases: properties more likely to find bugs are monitored in a critical phase to provide faster feedback to the developers; the rest are monitored in a background phase. We compare our techniques with the evolution-unaware (base) RV when monitoring test executions in 200 versions of 10 open-source projects. RPS and the RPP critical phase reduce the average RV overhead from 9.4x (for base RV) to 1.8x, without missing any new violations. VMS reduces the average number of violations 540x, from 54 violations per version (for base RV) to one violation per 10 versions.

KW - Regression testing

KW - Runtime verification

KW - Software evolution

KW - Software testing

KW - Specifications

UR - http://www.scopus.com/inward/record.url?scp=85067961352&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067961352&partnerID=8YFLogxK

U2 - 10.1109/ICST.2019.00037

DO - 10.1109/ICST.2019.00037

M3 - Conference contribution

AN - SCOPUS:85067961352

T3 - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

SP - 300

EP - 311

BT - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -