CrowdLearn: A Crowd-AI Hybrid System for Deep Learning-based Damage Assessment Applications

Daniel Yue Zhang, Yang Zhang, Qi Li, Thomas Plummer, Dong Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Artificial Intelligence (AI) has been widely adopted in many important application domains such as speech recognition, computer vision, autonomous driving, and AI for social good. In this paper, we focus on the AI-based damage assessment applications where deep neural network approaches are used to automatically identify damage severity of impacted areas from imagery reports in the aftermath of a disaster (e.g., earthquake, hurricane, landslides). While AI algorithms often significantly reduce the detection time and labor cost in such applications, their performance sometimes falls short of the desired accuracy and is considered to be less reliable than domain experts. To exacerbate the problem, the black-box nature of the AI algorithms also makes it difficult to troubleshoot the system when their performance is unsatisfactory. The emergence of crowdsourcing platforms (e.g., Amazon Mechanic Turk, Waze) brings about the opportunity to incorporate human intelligence into AI algorithms. However, the crowdsourcing platform is also black-box in terms of the uncertain response delay and crowd worker quality. In this work, we propose the CrowdLearn, a crowd-AI hybrid system that leverages the crowdsourcing platform to troubleshoot, tune, and eventually improve the black-box AI algorithms by welding crowd intelligence with machine intelligence. The system is specifically designed for deep learning-based damage assessment (DDA) applications where the crowd tend to be more accurate but less responsive than machines. Our evaluation results on a real-world case study on Amazon Mechanic Turk demonstrate that CrowdLearn can provide timely and more accurate assessments to natural disaster events than the state-of-the-art AI-only and human-AI integrated systems.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1221-1232
Number of pages12
ISBN (Electronic)9781728125190
DOIs
StatePublished - Jul 2019
Externally publishedYes
Event39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019 - Richardson, United States
Duration: Jul 7 2019Jul 9 2019

Publication series

NameProceedings - International Conference on Distributed Computing Systems
Volume2019-July

Conference

Conference39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019
Country/TerritoryUnited States
CityRichardson
Period7/7/197/9/19

Keywords

  • Damage Assessment
  • Deep Learning
  • Disaster Response
  • Human-AI system

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'CrowdLearn: A Crowd-AI Hybrid System for Deep Learning-based Damage Assessment Applications'. Together they form a unique fingerprint.

Cite this