Automated bridge inspection image interpretation based on vision-language pre-Training

Bridge inspection images capture a wealth of information and details about bridge conditions. This study proposes a method to interpret on-site bridge inspection images by generating human-readable descriptive sentences. The resulting text can be formed into a bridge inspection report to aid and expedite the bridge inspection process for bridge engineers; and the extracted information can be further exploited to support bridge deterioration prediction and maintenance decision making. This is, however, a challenging task that combines computer vision and natural language processing. First, it not only requires object detection/segmentation from the bridge inspection images but also demands a grasp of the relationships between the recognized objects. Second, human-readable sentences need to be generated based on the extracted information from the images. Third, the available bridge image-Text data pairs, which can be used for training, are quite limited and highly noisy. To address these gaps, this paper proposes a deep learning-based model for generating free-form human-readable descriptive sentences of the bridge conditions, which leverages bootstrapping language-image pre-Training (BLIP) and its vision-language pre-Training data from the web. This paper discusses the proposed model and its performance results.