An auxiliary attention-based network for joint classification and localization of breast tumor on ultrasound images

Zong Fan, Ping Gong, Xiaohui Zhang, Zhimin Wang, Yao Hao, Pengfei Song, Shigao Chen, Hua Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Ultrasound imaging is an effective screening tool for early diagnosis of breast tumor to decrease the mortality rate. However, differentiation of tumor type based on ultrasound images remains challenging in the field of medical imaging due to the inherent noise and speckles. Thus, obtaining additional information for lesion localization could better support the decision-making by clinicians and improve diagnosis fidelity. Recently, multi-task learning (MTL) methods have been proposed for joint tumor classification and localization, where promising results were demonstrated. However, most MTL methods trained independent network branches for the two different tasks, which might cause conflicts in optimizing features due to their different purposes. In addition, these methods usually require fully-segmented datasets for model training, which poses a heavy burden in data annotation. To overcome these limitations, we propose a novel MTL framework for joint breast tumor classification and localization, motivated by the idea of attention mechanism and weakly-supervised learning strategy. Our method has three major advantages. First, an auxiliary lesion-aware network (LA-Net) with multiple attention modules for lesion localization was designed on top of a pre-defined classification network. In this way, the extracted features for classification were directly augmented by the region of interest (ROI) predicted by the LA-Net, alleviating the potential conflicts between the two tasks. Second, a sequential training strategy with a weakly-supervised learning scheme was employed to train the LA-Net and the classification network iteratively, which allows the model to be trained on the partially-segmented datasets and reduces the burden on data annotation. Third, the LA-Net and classification network design are modularized so that both architectures can be flexibly adjusted for various applications. Results from experiments performed on two breast ultrasound image datasets demonstrated the effectiveness of the proposed method.

Original languageEnglish (US)
Title of host publicationMedical Imaging 2023
Subtitle of host publicationImage Processing
EditorsOlivier Colliot, Ivana Isgum
ISBN (Electronic)9781510660335
StatePublished - 2023
Externally publishedYes
EventMedical Imaging 2023: Image Processing - San Diego, United States
Duration: Feb 19 2023Feb 23 2023

Publication series

NameProgress in Biomedical Optics and Imaging - Proceedings of SPIE
ISSN (Print)1605-7422


ConferenceMedical Imaging 2023: Image Processing
Country/TerritoryUnited States
CitySan Diego

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Atomic and Molecular Physics, and Optics
  • Biomaterials
  • Radiology Nuclear Medicine and imaging


Dive into the research topics of 'An auxiliary attention-based network for joint classification and localization of breast tumor on ultrasound images'. Together they form a unique fingerprint.

Cite this