TY - GEN
T1 - Certifiable Evaluation for Autonomous Vehicle Perception Systems using Deep Importance Sampling (Deep IS)
AU - Arief, Mansur
AU - Cen, Zhepeng
AU - Liu, Zhenyuan
AU - Huang, Zhiyuan
AU - Li, Bo
AU - Lam, Henry
AU - Zhao, Ding
N1 - Publisher Copyright:
© 2022 IEEE.
PY - 2022
Y1 - 2022
N2 - Evaluating the performance of autonomous vehicles (AV) and their complex AI-driven functionalities to high precision under naturalistic conditions remains a challenge, especially when the failure or dangerous cases are rare. Rarity does not only require an enormous sample size for a naive method to achieve high confidence residual risk estimation, but it can also cause serious risk underestimation issues that is hard to detect. Meanwhile, the state-of-the-art rare safety-critical event evaluation approach that comes with a correctness guarantee can compute an upper bound for the true risk under certain conditions, which limits its practical uses. In this work, we propose Deep Importance Sampling (Deep IS) framework that utilizes a deep neural network to obtain an efficient less biased risk estimate, with an efficiency that is on par with that of the state-of-the-art method. In the numerical experiment evaluating the misclassification rate of a traffic sign classifier, Deep IS only needs 1/40-th of the samples required by a naive sampling method to achieve 10% relative error. Furthermore, the estimate produced by Deep IS is 10 times less conservative compared to the risk upper bound and only off by at most 10% difference to the true target. This efficient deep-learning-based IS procedure promises a highly efficient method to deal with often high-dimensional functional safety problems with rare naturalistic failure cases that are prevalent in AV domains.
AB - Evaluating the performance of autonomous vehicles (AV) and their complex AI-driven functionalities to high precision under naturalistic conditions remains a challenge, especially when the failure or dangerous cases are rare. Rarity does not only require an enormous sample size for a naive method to achieve high confidence residual risk estimation, but it can also cause serious risk underestimation issues that is hard to detect. Meanwhile, the state-of-the-art rare safety-critical event evaluation approach that comes with a correctness guarantee can compute an upper bound for the true risk under certain conditions, which limits its practical uses. In this work, we propose Deep Importance Sampling (Deep IS) framework that utilizes a deep neural network to obtain an efficient less biased risk estimate, with an efficiency that is on par with that of the state-of-the-art method. In the numerical experiment evaluating the misclassification rate of a traffic sign classifier, Deep IS only needs 1/40-th of the samples required by a naive sampling method to achieve 10% relative error. Furthermore, the estimate produced by Deep IS is 10 times less conservative compared to the risk upper bound and only off by at most 10% difference to the true target. This efficient deep-learning-based IS procedure promises a highly efficient method to deal with often high-dimensional functional safety problems with rare naturalistic failure cases that are prevalent in AV domains.
UR - http://www.scopus.com/inward/record.url?scp=85128579686&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85128579686&partnerID=8YFLogxK
U2 - 10.1109/ITSC55140.2022.9922202
DO - 10.1109/ITSC55140.2022.9922202
M3 - Conference contribution
AN - SCOPUS:85128579686
T3 - IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC
SP - 1736
EP - 1742
BT - 2022 IEEE 25th International Conference on Intelligent Transportation Systems, ITSC 2022
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 25th IEEE International Conference on Intelligent Transportation Systems, ITSC 2022
Y2 - 8 October 2022 through 12 October 2022
ER -