Boosted audio-visual HMM for speech reading

Pei Yin; Irfan Essa; James M. Rehg

Boosted audio-visual HMM for speech reading

Pei Yin, Irfan Essa, James M. Rehg

Research output: Contribution to journal › Conference article › peer-review

Abstract

We propose a new approach for combining acoustic and visual measurements to aid in recognizing lip shapes of a person speaking. Our method relies on computing the maximum likelihoods of (a) HMM used to model phonemes from the acoustic signal, and (b) HMM used to model visual features motions from video. One significant addition in this work is the dynamic analysis with features selected by Ada-Boost, on the basis of their discriminant ability. This form of integration, leading to boosted HMM, permits AdaBoost to find the best features first, and then uses HMM to exploit dynamic information inherent in the signal.

Original language	English (US)
Pages (from-to)	2013-2018
Number of pages	6
Journal	Conference Record of the Asilomar Conference on Signals, Systems and Computers
Volume	2
State	Published - 2003
Externally published	Yes
Event	Conference Record of the Thirty-Seventh Asilomar Conference on Signals, Systems and Computers - Pacific Grove, CA, United States Duration: Nov 9 2003 → Nov 12 2003

ASJC Scopus subject areas

Signal Processing
Computer Networks and Communications

Library availability

Discover UIUC Full Text

Cite this

@article{3efd9e1b98584ad7a23eff2287139a8c,

title = "Boosted audio-visual HMM for speech reading",

abstract = "We propose a new approach for combining acoustic and visual measurements to aid in recognizing lip shapes of a person speaking. Our method relies on computing the maximum likelihoods of (a) HMM used to model phonemes from the acoustic signal, and (b) HMM used to model visual features motions from video. One significant addition in this work is the dynamic analysis with features selected by Ada-Boost, on the basis of their discriminant ability. This form of integration, leading to boosted HMM, permits AdaBoost to find the best features first, and then uses HMM to exploit dynamic information inherent in the signal.",

author = "Pei Yin and Irfan Essa and Rehg, {James M.}",

year = "2003",

language = "English (US)",

volume = "2",

pages = "2013--2018",

journal = "Conference Record of the Asilomar Conference on Signals, Systems and Computers",

issn = "1058-6393",

publisher = "IEEE Computer Society",

note = "Conference Record of the Thirty-Seventh Asilomar Conference on Signals, Systems and Computers ; Conference date: 09-11-2003 Through 12-11-2003",

}

TY - JOUR

T1 - Boosted audio-visual HMM for speech reading

AU - Yin, Pei

AU - Essa, Irfan

AU - Rehg, James M.

PY - 2003

Y1 - 2003

N2 - We propose a new approach for combining acoustic and visual measurements to aid in recognizing lip shapes of a person speaking. Our method relies on computing the maximum likelihoods of (a) HMM used to model phonemes from the acoustic signal, and (b) HMM used to model visual features motions from video. One significant addition in this work is the dynamic analysis with features selected by Ada-Boost, on the basis of their discriminant ability. This form of integration, leading to boosted HMM, permits AdaBoost to find the best features first, and then uses HMM to exploit dynamic information inherent in the signal.

AB - We propose a new approach for combining acoustic and visual measurements to aid in recognizing lip shapes of a person speaking. Our method relies on computing the maximum likelihoods of (a) HMM used to model phonemes from the acoustic signal, and (b) HMM used to model visual features motions from video. One significant addition in this work is the dynamic analysis with features selected by Ada-Boost, on the basis of their discriminant ability. This form of integration, leading to boosted HMM, permits AdaBoost to find the best features first, and then uses HMM to exploit dynamic information inherent in the signal.

UR - http://www.scopus.com/inward/record.url?scp=4143078370&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=4143078370&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:4143078370

SN - 1058-6393

VL - 2

SP - 2013

EP - 2018

JO - Conference Record of the Asilomar Conference on Signals, Systems and Computers

JF - Conference Record of the Asilomar Conference on Signals, Systems and Computers

T2 - Conference Record of the Thirty-Seventh Asilomar Conference on Signals, Systems and Computers

Y2 - 9 November 2003 through 12 November 2003

ER -

Boosted audio-visual HMM for speech reading

Abstract

ASJC Scopus subject areas

Library availability

Related links

Fingerprint

Cite this