Storm: Program reduction for testing and debugging probabilistic programming systems

Saikat Dutta, Wenxian Zhang, Zixin Huang, Sasa Misailovic

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Probabilistic programming languages offer an intuitive way to model uncertainty by representing complex probability models as simple probabilistic programs. Probabilistic programming systems (PP systems) hide the complexity of inference algorithms away from the program developer. Unfortunately, if a failure occurs during the run of a PP system, a developer typically has very little support in finding the part of the probabilistic program that causes the failure in the system. This paper presents Storm, a novel general framework for reducing probabilistic programs. Given a probabilistic program (with associated data and inference arguments) that causes a failure in a PP system, Storm finds a smaller version of the program, data, and arguments that cause the same failure. Storm leverages both generic code and data transformations from compiler testing and domain-specific, probabilistic transformations. The paper presents new transformations that reduce the complexity of statements and expressions, reduce data size, and simplify inference arguments (e.g., the number of iterations of the inference algorithm). We evaluated Storm on 47 programs that caused failures in two popular probabilistic programming systems, Stan and Pyro. Our experimental results show Storms effectiveness. For Stan, our minimized programs have 49% less code, 67% less data, and 96% fewer iterations. For Pyro, our minimized programs have 58% less code, 96% less data, and 99% fewer iterations. We also show the benefits of Storm when debugging probabilistic programs.

Original languageEnglish (US)
Title of host publicationESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
EditorsSven Apel, Marlon Dumas, Alessandra Russo, Dietmar Pfahl
PublisherAssociation for Computing Machinery, Inc
Pages729-739
Number of pages11
ISBN (Electronic)9781450355728
DOIs
StatePublished - Aug 12 2019
Event27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019 - Tallinn, Estonia
Duration: Aug 26 2019Aug 30 2019

Publication series

NameESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Conference

Conference27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019
CountryEstonia
CityTallinn
Period8/26/198/30/19

Fingerprint

Computer debugging
Computer systems programming
Testing
Computer programming languages

Keywords

  • Probabilistic Programming Languages
  • Software Testing

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software

Cite this

Dutta, S., Zhang, W., Huang, Z., & Misailovic, S. (2019). Storm: Program reduction for testing and debugging probabilistic programming systems. In S. Apel, M. Dumas, A. Russo, & D. Pfahl (Eds.), ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering (pp. 729-739). (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering). Association for Computing Machinery, Inc. https://doi.org/10.1145/3338906.3338972

Storm : Program reduction for testing and debugging probabilistic programming systems. / Dutta, Saikat; Zhang, Wenxian; Huang, Zixin; Misailovic, Sasa.

ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ed. / Sven Apel; Marlon Dumas; Alessandra Russo; Dietmar Pfahl. Association for Computing Machinery, Inc, 2019. p. 729-739 (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Dutta, S, Zhang, W, Huang, Z & Misailovic, S 2019, Storm: Program reduction for testing and debugging probabilistic programming systems. in S Apel, M Dumas, A Russo & D Pfahl (eds), ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Association for Computing Machinery, Inc, pp. 729-739, 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, Tallinn, Estonia, 8/26/19. https://doi.org/10.1145/3338906.3338972
Dutta S, Zhang W, Huang Z, Misailovic S. Storm: Program reduction for testing and debugging probabilistic programming systems. In Apel S, Dumas M, Russo A, Pfahl D, editors, ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for Computing Machinery, Inc. 2019. p. 729-739. (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering). https://doi.org/10.1145/3338906.3338972
Dutta, Saikat ; Zhang, Wenxian ; Huang, Zixin ; Misailovic, Sasa. / Storm : Program reduction for testing and debugging probabilistic programming systems. ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. editor / Sven Apel ; Marlon Dumas ; Alessandra Russo ; Dietmar Pfahl. Association for Computing Machinery, Inc, 2019. pp. 729-739 (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering).
@inproceedings{ccf7ca9708e84f39b6132ea1d22ee72b,
title = "Storm: Program reduction for testing and debugging probabilistic programming systems",
abstract = "Probabilistic programming languages offer an intuitive way to model uncertainty by representing complex probability models as simple probabilistic programs. Probabilistic programming systems (PP systems) hide the complexity of inference algorithms away from the program developer. Unfortunately, if a failure occurs during the run of a PP system, a developer typically has very little support in finding the part of the probabilistic program that causes the failure in the system. This paper presents Storm, a novel general framework for reducing probabilistic programs. Given a probabilistic program (with associated data and inference arguments) that causes a failure in a PP system, Storm finds a smaller version of the program, data, and arguments that cause the same failure. Storm leverages both generic code and data transformations from compiler testing and domain-specific, probabilistic transformations. The paper presents new transformations that reduce the complexity of statements and expressions, reduce data size, and simplify inference arguments (e.g., the number of iterations of the inference algorithm). We evaluated Storm on 47 programs that caused failures in two popular probabilistic programming systems, Stan and Pyro. Our experimental results show Storms effectiveness. For Stan, our minimized programs have 49{\%} less code, 67{\%} less data, and 96{\%} fewer iterations. For Pyro, our minimized programs have 58{\%} less code, 96{\%} less data, and 99{\%} fewer iterations. We also show the benefits of Storm when debugging probabilistic programs.",
keywords = "Probabilistic Programming Languages, Software Testing",
author = "Saikat Dutta and Wenxian Zhang and Zixin Huang and Sasa Misailovic",
year = "2019",
month = "8",
day = "12",
doi = "10.1145/3338906.3338972",
language = "English (US)",
series = "ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering",
publisher = "Association for Computing Machinery, Inc",
pages = "729--739",
editor = "Sven Apel and Marlon Dumas and Alessandra Russo and Dietmar Pfahl",
booktitle = "ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering",

}

TY - GEN

T1 - Storm

T2 - Program reduction for testing and debugging probabilistic programming systems

AU - Dutta, Saikat

AU - Zhang, Wenxian

AU - Huang, Zixin

AU - Misailovic, Sasa

PY - 2019/8/12

Y1 - 2019/8/12

N2 - Probabilistic programming languages offer an intuitive way to model uncertainty by representing complex probability models as simple probabilistic programs. Probabilistic programming systems (PP systems) hide the complexity of inference algorithms away from the program developer. Unfortunately, if a failure occurs during the run of a PP system, a developer typically has very little support in finding the part of the probabilistic program that causes the failure in the system. This paper presents Storm, a novel general framework for reducing probabilistic programs. Given a probabilistic program (with associated data and inference arguments) that causes a failure in a PP system, Storm finds a smaller version of the program, data, and arguments that cause the same failure. Storm leverages both generic code and data transformations from compiler testing and domain-specific, probabilistic transformations. The paper presents new transformations that reduce the complexity of statements and expressions, reduce data size, and simplify inference arguments (e.g., the number of iterations of the inference algorithm). We evaluated Storm on 47 programs that caused failures in two popular probabilistic programming systems, Stan and Pyro. Our experimental results show Storms effectiveness. For Stan, our minimized programs have 49% less code, 67% less data, and 96% fewer iterations. For Pyro, our minimized programs have 58% less code, 96% less data, and 99% fewer iterations. We also show the benefits of Storm when debugging probabilistic programs.

AB - Probabilistic programming languages offer an intuitive way to model uncertainty by representing complex probability models as simple probabilistic programs. Probabilistic programming systems (PP systems) hide the complexity of inference algorithms away from the program developer. Unfortunately, if a failure occurs during the run of a PP system, a developer typically has very little support in finding the part of the probabilistic program that causes the failure in the system. This paper presents Storm, a novel general framework for reducing probabilistic programs. Given a probabilistic program (with associated data and inference arguments) that causes a failure in a PP system, Storm finds a smaller version of the program, data, and arguments that cause the same failure. Storm leverages both generic code and data transformations from compiler testing and domain-specific, probabilistic transformations. The paper presents new transformations that reduce the complexity of statements and expressions, reduce data size, and simplify inference arguments (e.g., the number of iterations of the inference algorithm). We evaluated Storm on 47 programs that caused failures in two popular probabilistic programming systems, Stan and Pyro. Our experimental results show Storms effectiveness. For Stan, our minimized programs have 49% less code, 67% less data, and 96% fewer iterations. For Pyro, our minimized programs have 58% less code, 96% less data, and 99% fewer iterations. We also show the benefits of Storm when debugging probabilistic programs.

KW - Probabilistic Programming Languages

KW - Software Testing

UR - http://www.scopus.com/inward/record.url?scp=85071906481&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071906481&partnerID=8YFLogxK

U2 - 10.1145/3338906.3338972

DO - 10.1145/3338906.3338972

M3 - Conference contribution

AN - SCOPUS:85071906481

T3 - ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering

SP - 729

EP - 739

BT - ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering

A2 - Apel, Sven

A2 - Dumas, Marlon

A2 - Russo, Alessandra

A2 - Pfahl, Dietmar

PB - Association for Computing Machinery, Inc

ER -