Revisiting Image-Language Networks for Open-Ended Phrase Detection

Bryan A. Plummer, Kevin J. Shih, Yichen Li, Ke Xu, Svetlana Lazebnik, Stan Sclaroff, Kate Saenko

Research output: Contribution to journalArticlepeer-review


Most existing work that grounds natural language phrases in images starts with the assumption that the phrase in question is relevant to the image. In this paper we address a more realistic version of the natural language grounding task where we must both identify whether the phrase is relevant to an image and localize the phrase. This can also be viewed as a generalization of object detection to an open-ended vocabulary, introducing elements of few- and zero-shot detection. We propose an approach for this task that extends Faster R-CNN to relate image regions and phrases. By carefully initializing the classification layers of our network using canonical correlation analysis (CCA), we encourage a solution that is more discerning when reasoning between similar phrases, resulting in over double the performance compared to a naive adaptation on three popular phrase grounding datasets, Flickr30K Entities, ReferIt Game, and Visual Genome, with test-time phrase vocabulary sizes of 5K, 32K, and 159K, respectively.

Original languageEnglish (US)
Pages (from-to)2155-2167
Number of pages13
JournalIEEE transactions on pattern analysis and machine intelligence
Issue number4
StatePublished - Apr 1 2022
Externally publishedYes


  • Object detection
  • Phrase grounding
  • Representation learning
  • Vision and language

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence
  • Applied Mathematics
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics


Dive into the research topics of 'Revisiting Image-Language Networks for Open-Ended Phrase Detection'. Together they form a unique fingerprint.

Cite this