Learned Hotelling Observers for use with Multi-Modal Data

Jason L. Granstedt, Fu Li, Umberto Villa, Mark A. Anastasio

Research output: Chapter in Book/Report/Conference proceedingConference contribution


In the medical imaging field, task-based metrics of image quality have been advocated as a mean to evaluate the performance of imaging systems and/or reconstruction algorithms. One such way of obtaining these metrics is through a numerical observer. Although the Bayesian ideal observer is optimal by definition, it is frequently intractable and nonlinear. Therefore, linear approximations to the IO are sometimes used to obtain task-based statistics. The optimal linear observer for maximizing the signal-To-noise ratio (SNR) of the test statistic is the Hotelling Observer (HO). However, the computational cost for obtaining the HO increases with image size and becomes intractable for large scale images. In multimodal data, this further becomes an issue because each additional modality dramatically increases the size of the composite image. An alternative to obtaining the HO is approximating the test statistic using a feed-forward neural network (FFNN). However, these methods of learning the HO have not been evaluated on multi-modal data. In this work, a tractable learned multi-modal observer is implemented. The considered task is a signal-known-statistically/background known statistically binary signal detection task. A stylized operator representing an ultrasound computed tomography imaging system and numerical breast phantoms with speed of sound and attenuation modalities are considered. The considered signal is a microcalcification cluster with a random amplitude. It is demonstrated that the learned HO can closely approximate the HO for the considered task. 2022 SPIE.

Original languageEnglish (US)
Title of host publicationMedical Imaging 2022
Subtitle of host publicationImage Perception, Observer Performance, and Technology Assessment
EditorsClaudia R. Mello-Thoms, Claudia R. Mello-Thoms, Sian Taylor-Phillips
ISBN (Electronic)9781510649453
StatePublished - 2022
EventMedical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment - Virtual, Online
Duration: Mar 21 2022Mar 27 2022

Publication series

NameProgress in Biomedical Optics and Imaging - Proceedings of SPIE
ISSN (Print)1605-7422


ConferenceMedical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment
CityVirtual, Online


  • Hotelling observer
  • Objective assessment of image quality
  • imaging system optimization
  • machine learning
  • multi-modal
  • neural networks
  • numerical observers
  • ultrasound computed tomography

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Atomic and Molecular Physics, and Optics
  • Biomaterials
  • Radiology Nuclear Medicine and imaging


Dive into the research topics of 'Learned Hotelling Observers for use with Multi-Modal Data'. Together they form a unique fingerprint.

Cite this