Read my lips: Brain dynamics associated with audiovisual integration and deviance detection

Chun Yu Tse, Gabriele Gratton, Susan M. Garnsey, Michael A. Novak, Monica Fabiani

Research output: Contribution to journalArticle

Abstract

Information from different modalities is initially processed in different brain areas, yet real-world perception often requires the integration of multisensory signals into a single percept. An example is the McGurk effect, in which people viewing a speaker whose lip movements do not match the utterance perceive the spoken sounds incorrectly, hearing them as more similar to those signaled by the visual rather than the auditory input. This indicates that audiovisual integration is important for generating the phoneme percept. Here we asked when and where the audiovisual integration process occurs, providing spatial and temporal boundaries for the processes generating phoneme perception. Specifically, we wanted to separate audiovisual integration from other processes, such as simple deviance detection. Building on previous work employing ERPs, we used an oddball paradigm in which task-irrelevant audiovisually deviant stimuli were embedded in strings of non-deviant stimuli. We also recorded the event-related optical signal, an imaging method combining spatial and temporal resolution, to investigate the time course and neuroanatomical substrate of audiovisual integration. We found that audiovisual deviants elicit a short duration response in the middle/superior temporal gyrus, whereas audiovisual integration elicits a more extended response involving also inferior frontal and occipital regions. Interactions between audiovisual integration and deviance detection processes were observed in the posterior/superior temporal gyrus. These data suggest that dynamic interactions between inferior frontal cortex and sensory regions play a significant role in multimodal integration.

Original languageEnglish (US)
Pages (from-to)1723-1737
Number of pages15
JournalJournal of Cognitive Neuroscience
Volume27
Issue number9
DOIs
StatePublished - Jan 1 2015

Fingerprint

Lip
Brain
Occipital Lobe
Optical Imaging
Frontal Lobe
Temporal Lobe
Hearing

ASJC Scopus subject areas

  • Cognitive Neuroscience

Cite this

Read my lips : Brain dynamics associated with audiovisual integration and deviance detection. / Tse, Chun Yu; Gratton, Gabriele; Garnsey, Susan M.; Novak, Michael A.; Fabiani, Monica.

In: Journal of Cognitive Neuroscience, Vol. 27, No. 9, 01.01.2015, p. 1723-1737.

Research output: Contribution to journalArticle

@article{021e467327db487bb96c9ad6b21c321e,
title = "Read my lips: Brain dynamics associated with audiovisual integration and deviance detection",
abstract = "Information from different modalities is initially processed in different brain areas, yet real-world perception often requires the integration of multisensory signals into a single percept. An example is the McGurk effect, in which people viewing a speaker whose lip movements do not match the utterance perceive the spoken sounds incorrectly, hearing them as more similar to those signaled by the visual rather than the auditory input. This indicates that audiovisual integration is important for generating the phoneme percept. Here we asked when and where the audiovisual integration process occurs, providing spatial and temporal boundaries for the processes generating phoneme perception. Specifically, we wanted to separate audiovisual integration from other processes, such as simple deviance detection. Building on previous work employing ERPs, we used an oddball paradigm in which task-irrelevant audiovisually deviant stimuli were embedded in strings of non-deviant stimuli. We also recorded the event-related optical signal, an imaging method combining spatial and temporal resolution, to investigate the time course and neuroanatomical substrate of audiovisual integration. We found that audiovisual deviants elicit a short duration response in the middle/superior temporal gyrus, whereas audiovisual integration elicits a more extended response involving also inferior frontal and occipital regions. Interactions between audiovisual integration and deviance detection processes were observed in the posterior/superior temporal gyrus. These data suggest that dynamic interactions between inferior frontal cortex and sensory regions play a significant role in multimodal integration.",
author = "Tse, {Chun Yu} and Gabriele Gratton and Garnsey, {Susan M.} and Novak, {Michael A.} and Monica Fabiani",
year = "2015",
month = "1",
day = "1",
doi = "10.1162/jocn_a_00812",
language = "English (US)",
volume = "27",
pages = "1723--1737",
journal = "Journal of Cognitive Neuroscience",
issn = "0898-929X",
publisher = "MIT Press Journals",
number = "9",

}

TY - JOUR

T1 - Read my lips

T2 - Brain dynamics associated with audiovisual integration and deviance detection

AU - Tse, Chun Yu

AU - Gratton, Gabriele

AU - Garnsey, Susan M.

AU - Novak, Michael A.

AU - Fabiani, Monica

PY - 2015/1/1

Y1 - 2015/1/1

N2 - Information from different modalities is initially processed in different brain areas, yet real-world perception often requires the integration of multisensory signals into a single percept. An example is the McGurk effect, in which people viewing a speaker whose lip movements do not match the utterance perceive the spoken sounds incorrectly, hearing them as more similar to those signaled by the visual rather than the auditory input. This indicates that audiovisual integration is important for generating the phoneme percept. Here we asked when and where the audiovisual integration process occurs, providing spatial and temporal boundaries for the processes generating phoneme perception. Specifically, we wanted to separate audiovisual integration from other processes, such as simple deviance detection. Building on previous work employing ERPs, we used an oddball paradigm in which task-irrelevant audiovisually deviant stimuli were embedded in strings of non-deviant stimuli. We also recorded the event-related optical signal, an imaging method combining spatial and temporal resolution, to investigate the time course and neuroanatomical substrate of audiovisual integration. We found that audiovisual deviants elicit a short duration response in the middle/superior temporal gyrus, whereas audiovisual integration elicits a more extended response involving also inferior frontal and occipital regions. Interactions between audiovisual integration and deviance detection processes were observed in the posterior/superior temporal gyrus. These data suggest that dynamic interactions between inferior frontal cortex and sensory regions play a significant role in multimodal integration.

AB - Information from different modalities is initially processed in different brain areas, yet real-world perception often requires the integration of multisensory signals into a single percept. An example is the McGurk effect, in which people viewing a speaker whose lip movements do not match the utterance perceive the spoken sounds incorrectly, hearing them as more similar to those signaled by the visual rather than the auditory input. This indicates that audiovisual integration is important for generating the phoneme percept. Here we asked when and where the audiovisual integration process occurs, providing spatial and temporal boundaries for the processes generating phoneme perception. Specifically, we wanted to separate audiovisual integration from other processes, such as simple deviance detection. Building on previous work employing ERPs, we used an oddball paradigm in which task-irrelevant audiovisually deviant stimuli were embedded in strings of non-deviant stimuli. We also recorded the event-related optical signal, an imaging method combining spatial and temporal resolution, to investigate the time course and neuroanatomical substrate of audiovisual integration. We found that audiovisual deviants elicit a short duration response in the middle/superior temporal gyrus, whereas audiovisual integration elicits a more extended response involving also inferior frontal and occipital regions. Interactions between audiovisual integration and deviance detection processes were observed in the posterior/superior temporal gyrus. These data suggest that dynamic interactions between inferior frontal cortex and sensory regions play a significant role in multimodal integration.

UR - http://www.scopus.com/inward/record.url?scp=84944452586&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84944452586&partnerID=8YFLogxK

U2 - 10.1162/jocn_a_00812

DO - 10.1162/jocn_a_00812

M3 - Article

C2 - 25848682

AN - SCOPUS:84944452586

VL - 27

SP - 1723

EP - 1737

JO - Journal of Cognitive Neuroscience

JF - Journal of Cognitive Neuroscience

SN - 0898-929X

IS - 9

ER -