Certifiable Evaluation for Autonomous Vehicle Perception Systems using Deep Importance Sampling (Deep IS)

Mansur Arief, Zhepeng Cen, Zhenyuan Liu, Zhiyuan Huang, Bo Li, Henry Lam, Ding Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Evaluating the performance of autonomous vehicles (AV) and their complex AI-driven functionalities to high precision under naturalistic conditions remains a challenge, especially when the failure or dangerous cases are rare. Rarity does not only require an enormous sample size for a naive method to achieve high confidence residual risk estimation, but it can also cause serious risk underestimation issues that is hard to detect. Meanwhile, the state-of-the-art rare safety-critical event evaluation approach that comes with a correctness guarantee can compute an upper bound for the true risk under certain conditions, which limits its practical uses. In this work, we propose Deep Importance Sampling (Deep IS) framework that utilizes a deep neural network to obtain an efficient less biased risk estimate, with an efficiency that is on par with that of the state-of-the-art method. In the numerical experiment evaluating the misclassification rate of a traffic sign classifier, Deep IS only needs 1/40-th of the samples required by a naive sampling method to achieve 10% relative error. Furthermore, the estimate produced by Deep IS is 10 times less conservative compared to the risk upper bound and only off by at most 10% difference to the true target. This efficient deep-learning-based IS procedure promises a highly efficient method to deal with often high-dimensional functional safety problems with rare naturalistic failure cases that are prevalent in AV domains.

Original languageEnglish (US)
Title of host publication2022 IEEE 25th International Conference on Intelligent Transportation Systems, ITSC 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1736-1742
Number of pages7
ISBN (Electronic)9781665468800
DOIs
StatePublished - 2022
Event25th IEEE International Conference on Intelligent Transportation Systems, ITSC 2022 - Macau, China
Duration: Oct 8 2022Oct 12 2022

Publication series

NameIEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC
Volume2022-October

Conference

Conference25th IEEE International Conference on Intelligent Transportation Systems, ITSC 2022
Country/TerritoryChina
CityMacau
Period10/8/2210/12/22

ASJC Scopus subject areas

  • Automotive Engineering
  • Mechanical Engineering
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'Certifiable Evaluation for Autonomous Vehicle Perception Systems using Deep Importance Sampling (Deep IS)'. Together they form a unique fingerprint.

Cite this