Supporting audiovisual query using dynamic programming

M. R. Naphade, R. Wang, T. S. Huang

Research output: Contribution to conferencePaperpeer-review


A necessary capability for content-based retrieval is to support the paradigm of query by example. Most systems for video retrieval support queries using image sequences only. We present an algorithm for matching multimodal (audio-visual) patterns for the purpose of content-based video retrieval. The novel ability of our approach to use the information content in multiple media coupled with a strong emphasis on temporal similarity differentiates it from the state-of-the-art in content-based retrieval. At the core of the pattern matching scheme is a dynamic programming algorithm, which leads to a significant improvement in performance. Coupling the use of audio with video this algorithm can be applied to grouping of shots based on audio-visual similarity. We also support relevance feedback. The user can provide feedback to the system, by choosing clips, which are closer to the user's desired target. The system then automatically adjusts the relative weights or relevance of the media and fetches different sets of target clips accordingly. It is our observation that a few iterations of such feedback are generally sufficient, for retrieving the desired video clips.

Original languageEnglish (US)
Number of pages10
StatePublished - 2001
Event-ACM Multimedia 2001 Workshops- 2001 Multimedia Conference - Ottawa, Ont., Canada
Duration: Sep 30 2001Oct 5 2001


Other-ACM Multimedia 2001 Workshops- 2001 Multimedia Conference
CityOttawa, Ont.


  • Dynamic programming
  • Nonlinear warping
  • Relevance feedback
  • Video retrieval

ASJC Scopus subject areas

  • General Computer Science


Dive into the research topics of 'Supporting audiovisual query using dynamic programming'. Together they form a unique fingerprint.

Cite this