Modeling the statistics of image features and associated text

Kobus Barnard, Pinar Duygulu, David Forsyth

Research output: Contribution to journalConference article

Abstract

We present a methodology for modeling the statistics of image features and associated text in large datasets. The models used also serve to cluster the images, as images are modeled as being produced by sampling from a limited number of combinations of mixing components. Furthermore, because our approach models the joint occurrence image features and associated text, it can be used to predict the occurrence of either, based on observations or queries. This supports an attractive approach to image search as well as novel applications such a suggesting illustrations for blocks of text (auto-illustrate) and generating words for images outside the training set (auto-annotate). In this paper we illustrate the approach on 10,000 images of work from the Fine Arts Museum of San Francisco. The images include line drawings, paintings, and pictures of sculpture and ceramics. Many of the images have associated free text whose nature varies greatly, from physical description to interpretation and mood. We incorporate statistical natural language processing in order to deal with free text. We use WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships.

Original languageEnglish (US)
Pages (from-to)1-11
Number of pages11
JournalProceedings of SPIE - The International Society for Optical Engineering
Volume4670
DOIs
StatePublished - Jan 1 2002
Externally publishedYes
EventDocumentation Recognition and Retrieval IX - San Jose, CA, United States
Duration: Jan 21 2002Jan 22 2002

Fingerprint

Semantics
Statistics
statistics
Museums
Painting
Modeling
Sampling
Processing
semantics
natural language processing
moods
occurrences
Text
Line Drawing
museums
Mood
WordNet
arts
Large Data Sets
Grouping

Keywords

  • Aspect model
  • Hierarchical clustering
  • Image retrieval
  • Learning image semantics
  • Recognition

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering

Cite this

Modeling the statistics of image features and associated text. / Barnard, Kobus; Duygulu, Pinar; Forsyth, David.

In: Proceedings of SPIE - The International Society for Optical Engineering, Vol. 4670, 01.01.2002, p. 1-11.

Research output: Contribution to journalConference article

@article{7685f8d9bca24b828cd066f4a67436ab,
title = "Modeling the statistics of image features and associated text",
abstract = "We present a methodology for modeling the statistics of image features and associated text in large datasets. The models used also serve to cluster the images, as images are modeled as being produced by sampling from a limited number of combinations of mixing components. Furthermore, because our approach models the joint occurrence image features and associated text, it can be used to predict the occurrence of either, based on observations or queries. This supports an attractive approach to image search as well as novel applications such a suggesting illustrations for blocks of text (auto-illustrate) and generating words for images outside the training set (auto-annotate). In this paper we illustrate the approach on 10,000 images of work from the Fine Arts Museum of San Francisco. The images include line drawings, paintings, and pictures of sculpture and ceramics. Many of the images have associated free text whose nature varies greatly, from physical description to interpretation and mood. We incorporate statistical natural language processing in order to deal with free text. We use WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships.",
keywords = "Aspect model, Hierarchical clustering, Image retrieval, Learning image semantics, Recognition",
author = "Kobus Barnard and Pinar Duygulu and David Forsyth",
year = "2002",
month = "1",
day = "1",
doi = "10.1117/12.450716",
language = "English (US)",
volume = "4670",
pages = "1--11",
journal = "Proceedings of SPIE - The International Society for Optical Engineering",
issn = "0277-786X",
publisher = "SPIE",

}

TY - JOUR

T1 - Modeling the statistics of image features and associated text

AU - Barnard, Kobus

AU - Duygulu, Pinar

AU - Forsyth, David

PY - 2002/1/1

Y1 - 2002/1/1

N2 - We present a methodology for modeling the statistics of image features and associated text in large datasets. The models used also serve to cluster the images, as images are modeled as being produced by sampling from a limited number of combinations of mixing components. Furthermore, because our approach models the joint occurrence image features and associated text, it can be used to predict the occurrence of either, based on observations or queries. This supports an attractive approach to image search as well as novel applications such a suggesting illustrations for blocks of text (auto-illustrate) and generating words for images outside the training set (auto-annotate). In this paper we illustrate the approach on 10,000 images of work from the Fine Arts Museum of San Francisco. The images include line drawings, paintings, and pictures of sculpture and ceramics. Many of the images have associated free text whose nature varies greatly, from physical description to interpretation and mood. We incorporate statistical natural language processing in order to deal with free text. We use WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships.

AB - We present a methodology for modeling the statistics of image features and associated text in large datasets. The models used also serve to cluster the images, as images are modeled as being produced by sampling from a limited number of combinations of mixing components. Furthermore, because our approach models the joint occurrence image features and associated text, it can be used to predict the occurrence of either, based on observations or queries. This supports an attractive approach to image search as well as novel applications such a suggesting illustrations for blocks of text (auto-illustrate) and generating words for images outside the training set (auto-annotate). In this paper we illustrate the approach on 10,000 images of work from the Fine Arts Museum of San Francisco. The images include line drawings, paintings, and pictures of sculpture and ceramics. Many of the images have associated free text whose nature varies greatly, from physical description to interpretation and mood. We incorporate statistical natural language processing in order to deal with free text. We use WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships.

KW - Aspect model

KW - Hierarchical clustering

KW - Image retrieval

KW - Learning image semantics

KW - Recognition

UR - http://www.scopus.com/inward/record.url?scp=0036031157&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0036031157&partnerID=8YFLogxK

U2 - 10.1117/12.450716

DO - 10.1117/12.450716

M3 - Conference article

AN - SCOPUS:0036031157

VL - 4670

SP - 1

EP - 11

JO - Proceedings of SPIE - The International Society for Optical Engineering

JF - Proceedings of SPIE - The International Society for Optical Engineering

SN - 0277-786X

ER -