Failure-adverse closed-loop statistical verification

The increasing complexity of control systems highlights the importance of verification analysis to ensure the resulting closed-loop system robustly satisfies necessary performance requirements. Safety-critical performance requirements present further challenges as failure to satisfy these requirements have severe real-world consequences. For physical systems such as cars or aircraft, simulation-based analysis provides a good indication of the real-world system’s performance without risking physical property or people, but cannot completely replace experimental testing due to discrepancies between simulation models and the real world. Given these considerations, this paper develops new statistical verification frameworks for failure-adverse testing of safety-critical performance requirements in experimental domains. Extensions of recently-developed closed-loop statistical verification algorithms form the core of this failure-adverse approach. These new failure-adverse algorithms break verification into a simulation stage and a hardware stage and carefully select experimental tests so as to maximize the information gained with each experiment while minimizing the likelihood of encountering failures. Two case studies compare the failure-adverse closed-loop verification approach against existing verification techniques and demonstrate its effectiveness in substantially decreasing the number of failures without sacrifices in prediction accuracy.