TY - GEN
T1 - Deephunter
T2 - 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019
AU - Xie, Xiaofei
AU - Ma, Lei
AU - Juefei-Xu, Felix
AU - Xue, Minhui
AU - Chen, Hongxu
AU - Liu, Yang
AU - Zhao, Jianjun
AU - Li, Bo
AU - Yin, Jianxiong
AU - See, Simon
PY - 2019/7/10
Y1 - 2019/7/10
N2 - The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a seed selection strategy that combines both diversity-based and recency-based seed selection. We implement and incorporate 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) our metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by up to a 98% validity ratio; (2) the diversity-based seed selection generally weighs more than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter outperforms the state of the arts by coverage as well as the quantity and diversity of defects identified; (4) guided by corner-region based criteria, DeepHunter is useful to capture defects during the DNN quantization for platform migration.
AB - The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a seed selection strategy that combines both diversity-based and recency-based seed selection. We implement and incorporate 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) our metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by up to a 98% validity ratio; (2) the diversity-based seed selection generally weighs more than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter outperforms the state of the arts by coverage as well as the quantity and diversity of defects identified; (4) guided by corner-region based criteria, DeepHunter is useful to capture defects during the DNN quantization for platform migration.
KW - Coverage-guided fuzzing
KW - Deep learning testing
KW - Metamorphic testing
UR - http://www.scopus.com/inward/record.url?scp=85070586358&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85070586358&partnerID=8YFLogxK
U2 - 10.1145/3293882.3330579
DO - 10.1145/3293882.3330579
M3 - Conference contribution
AN - SCOPUS:85070586358
T3 - ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis
SP - 158
EP - 168
BT - ISSTA 2019 - Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis
A2 - Zhang, Dongmei
A2 - Moller, Anders
PB - Association for Computing Machinery, Inc
Y2 - 15 July 2019 through 19 July 2019
ER -