Deep learning-based damage detection of miter gates using synthetic imagery from computer graphics

Vedhus Hoskere, Yasutaka Narazaki, Billie F. Spencer, Matthew D. Smith

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Structural inspections of large, difficult-to-access infrastructure like dams and bridges are often time-consuming, laborious and unsafe. In the United States, federal and state agencies responsible for managing such infrastructure assets are investigating the use of unmanned aerial vehicles (UAV) to allow for remote data acquisition. Processing the large amounts of data acquired by the UAV remains a challenging task. Over the past four years, researchers have been investigating deep learning methods for automated damage detection through image classification and more recently, the use of semantic segmentation where each pixel in the image is given a certain label. For such algorithms to work effectively, deep neural networks need to be trained on large datasets of labelled images. The generation of these labels for semantic segmentation is a very tedious process as it requires each pixel in the image to be labelled. This paper investigates the use of computer graphics to automatically generate synthetic imagery for the purposes of training deep learning algorithms for vision-based damage detection using semantic segmentation. The significant advantage of this is the automatic generation of precise semantic labels due to the implicit information in the developed graphics models. Parametric noise-based graphics texture models are created for defects such as cracks and corrosion and for other features such as vegetation growth, and dirt. The parameterization of the texture models allows for generation of a range of different surface conditions, thereby providing increased flexibility over data generation. To demonstrate the benefits of the proposed methodology for synthetic data generation a virtual environment of inland navigation infrastructure including miter gates and tainter gate dams is created. The developed texture models are applied to the virtual environment to produce a photo-realistic model. Synthetic image data is then rendered from the developed model and used to demonstrate the efficacy for training deep learning-based semantic segmentation algorithms for damage detection.

Original languageEnglish (US)
Title of host publicationStructural Health Monitoring 2019
Subtitle of host publicationEnabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring
EditorsFu-Kuo Chang, Alfredo Guemes, Fotis Kopsaftopoulos
PublisherDEStech Publications Inc.
Pages3073-3080
Number of pages8
ISBN (Electronic)9781605956015
StatePublished - Jan 1 2019
Event12th International Workshop on Structural Health Monitoring: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT), IWSHM 2019 - Stanford, United States
Duration: Sep 10 2019Sep 12 2019

Publication series

NameStructural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring
Volume2

Conference

Conference12th International Workshop on Structural Health Monitoring: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT), IWSHM 2019
CountryUnited States
CityStanford
Period9/10/199/12/19

Fingerprint

Computer Graphics
Damage detection
Imagery (Psychotherapy)
Computer graphics
Semantics
Learning
Labels
Textures
Unmanned aerial vehicles (UAV)
Virtual reality
Dams
Pixels
Corrosion
Image classification
Noise
Parameterization
Learning algorithms
Research Personnel
Deep learning
Data acquisition

ASJC Scopus subject areas

  • Computer Science Applications
  • Health Information Management

Cite this

Hoskere, V., Narazaki, Y., Spencer, B. F., & Smith, M. D. (2019). Deep learning-based damage detection of miter gates using synthetic imagery from computer graphics. In F-K. Chang, A. Guemes, & F. Kopsaftopoulos (Eds.), Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring (pp. 3073-3080). (Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring; Vol. 2). DEStech Publications Inc..

Deep learning-based damage detection of miter gates using synthetic imagery from computer graphics. / Hoskere, Vedhus; Narazaki, Yasutaka; Spencer, Billie F.; Smith, Matthew D.

Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring. ed. / Fu-Kuo Chang; Alfredo Guemes; Fotis Kopsaftopoulos. DEStech Publications Inc., 2019. p. 3073-3080 (Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring; Vol. 2).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Hoskere, V, Narazaki, Y, Spencer, BF & Smith, MD 2019, Deep learning-based damage detection of miter gates using synthetic imagery from computer graphics. in F-K Chang, A Guemes & F Kopsaftopoulos (eds), Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring. Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring, vol. 2, DEStech Publications Inc., pp. 3073-3080, 12th International Workshop on Structural Health Monitoring: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT), IWSHM 2019, Stanford, United States, 9/10/19.
Hoskere V, Narazaki Y, Spencer BF, Smith MD. Deep learning-based damage detection of miter gates using synthetic imagery from computer graphics. In Chang F-K, Guemes A, Kopsaftopoulos F, editors, Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring. DEStech Publications Inc. 2019. p. 3073-3080. (Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring).
Hoskere, Vedhus ; Narazaki, Yasutaka ; Spencer, Billie F. ; Smith, Matthew D. / Deep learning-based damage detection of miter gates using synthetic imagery from computer graphics. Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring. editor / Fu-Kuo Chang ; Alfredo Guemes ; Fotis Kopsaftopoulos. DEStech Publications Inc., 2019. pp. 3073-3080 (Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring).
@inproceedings{e9943956f230422281e492d8bb34f43b,
title = "Deep learning-based damage detection of miter gates using synthetic imagery from computer graphics",
abstract = "Structural inspections of large, difficult-to-access infrastructure like dams and bridges are often time-consuming, laborious and unsafe. In the United States, federal and state agencies responsible for managing such infrastructure assets are investigating the use of unmanned aerial vehicles (UAV) to allow for remote data acquisition. Processing the large amounts of data acquired by the UAV remains a challenging task. Over the past four years, researchers have been investigating deep learning methods for automated damage detection through image classification and more recently, the use of semantic segmentation where each pixel in the image is given a certain label. For such algorithms to work effectively, deep neural networks need to be trained on large datasets of labelled images. The generation of these labels for semantic segmentation is a very tedious process as it requires each pixel in the image to be labelled. This paper investigates the use of computer graphics to automatically generate synthetic imagery for the purposes of training deep learning algorithms for vision-based damage detection using semantic segmentation. The significant advantage of this is the automatic generation of precise semantic labels due to the implicit information in the developed graphics models. Parametric noise-based graphics texture models are created for defects such as cracks and corrosion and for other features such as vegetation growth, and dirt. The parameterization of the texture models allows for generation of a range of different surface conditions, thereby providing increased flexibility over data generation. To demonstrate the benefits of the proposed methodology for synthetic data generation a virtual environment of inland navigation infrastructure including miter gates and tainter gate dams is created. The developed texture models are applied to the virtual environment to produce a photo-realistic model. Synthetic image data is then rendered from the developed model and used to demonstrate the efficacy for training deep learning-based semantic segmentation algorithms for damage detection.",
author = "Vedhus Hoskere and Yasutaka Narazaki and Spencer, {Billie F.} and Smith, {Matthew D.}",
year = "2019",
month = "1",
day = "1",
language = "English (US)",
series = "Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring",
publisher = "DEStech Publications Inc.",
pages = "3073--3080",
editor = "Fu-Kuo Chang and Alfredo Guemes and Fotis Kopsaftopoulos",
booktitle = "Structural Health Monitoring 2019",

}

TY - GEN

T1 - Deep learning-based damage detection of miter gates using synthetic imagery from computer graphics

AU - Hoskere, Vedhus

AU - Narazaki, Yasutaka

AU - Spencer, Billie F.

AU - Smith, Matthew D.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Structural inspections of large, difficult-to-access infrastructure like dams and bridges are often time-consuming, laborious and unsafe. In the United States, federal and state agencies responsible for managing such infrastructure assets are investigating the use of unmanned aerial vehicles (UAV) to allow for remote data acquisition. Processing the large amounts of data acquired by the UAV remains a challenging task. Over the past four years, researchers have been investigating deep learning methods for automated damage detection through image classification and more recently, the use of semantic segmentation where each pixel in the image is given a certain label. For such algorithms to work effectively, deep neural networks need to be trained on large datasets of labelled images. The generation of these labels for semantic segmentation is a very tedious process as it requires each pixel in the image to be labelled. This paper investigates the use of computer graphics to automatically generate synthetic imagery for the purposes of training deep learning algorithms for vision-based damage detection using semantic segmentation. The significant advantage of this is the automatic generation of precise semantic labels due to the implicit information in the developed graphics models. Parametric noise-based graphics texture models are created for defects such as cracks and corrosion and for other features such as vegetation growth, and dirt. The parameterization of the texture models allows for generation of a range of different surface conditions, thereby providing increased flexibility over data generation. To demonstrate the benefits of the proposed methodology for synthetic data generation a virtual environment of inland navigation infrastructure including miter gates and tainter gate dams is created. The developed texture models are applied to the virtual environment to produce a photo-realistic model. Synthetic image data is then rendered from the developed model and used to demonstrate the efficacy for training deep learning-based semantic segmentation algorithms for damage detection.

AB - Structural inspections of large, difficult-to-access infrastructure like dams and bridges are often time-consuming, laborious and unsafe. In the United States, federal and state agencies responsible for managing such infrastructure assets are investigating the use of unmanned aerial vehicles (UAV) to allow for remote data acquisition. Processing the large amounts of data acquired by the UAV remains a challenging task. Over the past four years, researchers have been investigating deep learning methods for automated damage detection through image classification and more recently, the use of semantic segmentation where each pixel in the image is given a certain label. For such algorithms to work effectively, deep neural networks need to be trained on large datasets of labelled images. The generation of these labels for semantic segmentation is a very tedious process as it requires each pixel in the image to be labelled. This paper investigates the use of computer graphics to automatically generate synthetic imagery for the purposes of training deep learning algorithms for vision-based damage detection using semantic segmentation. The significant advantage of this is the automatic generation of precise semantic labels due to the implicit information in the developed graphics models. Parametric noise-based graphics texture models are created for defects such as cracks and corrosion and for other features such as vegetation growth, and dirt. The parameterization of the texture models allows for generation of a range of different surface conditions, thereby providing increased flexibility over data generation. To demonstrate the benefits of the proposed methodology for synthetic data generation a virtual environment of inland navigation infrastructure including miter gates and tainter gate dams is created. The developed texture models are applied to the virtual environment to produce a photo-realistic model. Synthetic image data is then rendered from the developed model and used to demonstrate the efficacy for training deep learning-based semantic segmentation algorithms for damage detection.

UR - http://www.scopus.com/inward/record.url?scp=85074298013&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85074298013&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85074298013

T3 - Structural Health Monitoring 2019: Enabling Intelligent Life-Cycle Health Management for Industry Internet of Things (IIOT) - Proceedings of the 12th International Workshop on Structural Health Monitoring

SP - 3073

EP - 3080

BT - Structural Health Monitoring 2019

A2 - Chang, Fu-Kuo

A2 - Guemes, Alfredo

A2 - Kopsaftopoulos, Fotis

PB - DEStech Publications Inc.

ER -