Storm: Program reduction for testing and debugging probabilistic programming systems

Saikat Dutta, Wenxian Zhang, Zixin Huang, Sasa Misailovic

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Probabilistic programming languages offer an intuitive way to model uncertainty by representing complex probability models as simple probabilistic programs. Probabilistic programming systems (PP systems) hide the complexity of inference algorithms away from the program developer. Unfortunately, if a failure occurs during the run of a PP system, a developer typically has very little support in finding the part of the probabilistic program that causes the failure in the system. This paper presents Storm, a novel general framework for reducing probabilistic programs. Given a probabilistic program (with associated data and inference arguments) that causes a failure in a PP system, Storm finds a smaller version of the program, data, and arguments that cause the same failure. Storm leverages both generic code and data transformations from compiler testing and domain-specific, probabilistic transformations. The paper presents new transformations that reduce the complexity of statements and expressions, reduce data size, and simplify inference arguments (e.g., the number of iterations of the inference algorithm). We evaluated Storm on 47 programs that caused failures in two popular probabilistic programming systems, Stan and Pyro. Our experimental results show Storms effectiveness. For Stan, our minimized programs have 49% less code, 67% less data, and 96% fewer iterations. For Pyro, our minimized programs have 58% less code, 96% less data, and 99% fewer iterations. We also show the benefits of Storm when debugging probabilistic programs.

Original languageEnglish (US)
Title of host publicationESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
EditorsSven Apel, Marlon Dumas, Alessandra Russo, Dietmar Pfahl
PublisherAssociation for Computing Machinery, Inc
Pages729-739
Number of pages11
ISBN (Electronic)9781450355728
DOIs
StatePublished - Aug 12 2019
Event27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019 - Tallinn, Estonia
Duration: Aug 26 2019Aug 30 2019

Publication series

NameESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Conference

Conference27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019
CountryEstonia
CityTallinn
Period8/26/198/30/19

Keywords

  • Probabilistic Programming Languages
  • Software Testing

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software

Fingerprint Dive into the research topics of 'Storm: Program reduction for testing and debugging probabilistic programming systems'. Together they form a unique fingerprint.

  • Cite this

    Dutta, S., Zhang, W., Huang, Z., & Misailovic, S. (2019). Storm: Program reduction for testing and debugging probabilistic programming systems. In S. Apel, M. Dumas, A. Russo, & D. Pfahl (Eds.), ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering (pp. 729-739). (ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering). Association for Computing Machinery, Inc. https://doi.org/10.1145/3338906.3338972