Learning deep structure-preserving image-text embeddings

Liwei Wang; Yin Li; Svetlana Lazebnik

doi:10.1109/CVPR.2016.541

Learning deep structure-preserving image-text embeddings

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a largemargin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and textto-image retrieval. Our method achieves new state-of-theart results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.

Original language	English (US)
Title of host publication	Proceedings - 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016
Publisher	IEEE Computer Society
Pages	5005-5013
Number of pages	9
ISBN (Electronic)	9781467388504
DOIs	https://doi.org/10.1109/CVPR.2016.541
State	Published - Dec 9 2016
Event	29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 - Las Vegas, United States Duration: Jun 26 2016 → Jul 1 2016

Publication series

Name	Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume	2016-December
ISSN (Print)	1063-6919

Conference

Conference	29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016
Country/Territory	United States
City	Las Vegas
Period	6/26/16 → 7/1/16

ASJC Scopus subject areas

Software
Computer Vision and Pattern Recognition

Online availability

10.1109/CVPR.2016.541

Library availability

Discover UIUC Full Text

Cite this

Wang, L., Li, Y., & Lazebnik, S. (2016). Learning deep structure-preserving image-text embeddings. In Proceedings - 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 (pp. 5005-5013). Article 7780910 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 2016-December). IEEE Computer Society. https://doi.org/10.1109/CVPR.2016.541

Learning deep structure-preserving image-text embeddings. / Wang, Liwei; Li, Yin; Lazebnik, Svetlana.
Proceedings - 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016. IEEE Computer Society, 2016. p. 5005-5013 7780910 (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Vol. 2016-December).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Wang, L, Li, Y & Lazebnik, S 2016, Learning deep structure-preserving image-text embeddings. in Proceedings - 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016., 7780910, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2016-December, IEEE Computer Society, pp. 5005-5013, 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, United States, 6/26/16. https://doi.org/10.1109/CVPR.2016.541

@inproceedings{1cfae09bc1b1411cace05d9ff54ebc29,

title = "Learning deep structure-preserving image-text embeddings",

abstract = "This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a largemargin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and textto-image retrieval. Our method achieves new state-of-theart results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.",

author = "Liwei Wang and Yin Li and Svetlana Lazebnik",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 ; Conference date: 26-06-2016 Through 01-07-2016",

year = "2016",

month = dec,

day = "9",

doi = "10.1109/CVPR.2016.541",

language = "English (US)",

series = "Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition",

publisher = "IEEE Computer Society",

pages = "5005--5013",

booktitle = "Proceedings - 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016",

}

TY - GEN

T1 - Learning deep structure-preserving image-text embeddings

AU - Wang, Liwei

AU - Li, Yin

AU - Lazebnik, Svetlana

PY - 2016/12/9

Y1 - 2016/12/9

N2 - This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a largemargin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and textto-image retrieval. Our method achieves new state-of-theart results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.

AB - This paper proposes a method for learning joint embeddings of images and text using a two-branch neural network with multiple layers of linear projections followed by nonlinearities. The network is trained using a largemargin objective that combines cross-view ranking constraints with within-view neighborhood structure preservation constraints inspired by metric learning literature. Extensive experiments show that our approach gains significant improvements in accuracy for image-to-text and textto-image retrieval. Our method achieves new state-of-theart results on the Flickr30K and MSCOCO image-sentence datasets and shows promise on the new task of phrase localization on the Flickr30K Entities dataset.

UR - http://www.scopus.com/inward/record.url?scp=84986271102&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84986271102&partnerID=8YFLogxK

U2 - 10.1109/CVPR.2016.541

DO - 10.1109/CVPR.2016.541

M3 - Conference contribution

AN - SCOPUS:84986271102

T3 - Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

SP - 5005

EP - 5013

BT - Proceedings - 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016

PB - IEEE Computer Society

T2 - 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016

Y2 - 26 June 2016 through 1 July 2016

ER -

Learning deep structure-preserving image-text embeddings

Abstract

Publication series

Conference

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this