Read my lips: Brain dynamics associated with audiovisual integration and deviance detection

Chun Yu Tse, Gabriele Gratton, Susan M. Garnsey, Michael A. Novak, Monica Fabiani

Research output: Contribution to journalArticle

Abstract

Information from different modalities is initially processed in different brain areas, yet real-world perception often requires the integration of multisensory signals into a single percept. An example is the McGurk effect, in which people viewing a speaker whose lip movements do not match the utterance perceive the spoken sounds incorrectly, hearing them as more similar to those signaled by the visual rather than the auditory input. This indicates that audiovisual integration is important for generating the phoneme percept. Here we asked when and where the audiovisual integration process occurs, providing spatial and temporal boundaries for the processes generating phoneme perception. Specifically, we wanted to separate audiovisual integration from other processes, such as simple deviance detection. Building on previous work employing ERPs, we used an oddball paradigm in which task-irrelevant audiovisually deviant stimuli were embedded in strings of non-deviant stimuli. We also recorded the event-related optical signal, an imaging method combining spatial and temporal resolution, to investigate the time course and neuroanatomical substrate of audiovisual integration. We found that audiovisual deviants elicit a short duration response in the middle/superior temporal gyrus, whereas audiovisual integration elicits a more extended response involving also inferior frontal and occipital regions. Interactions between audiovisual integration and deviance detection processes were observed in the posterior/superior temporal gyrus. These data suggest that dynamic interactions between inferior frontal cortex and sensory regions play a significant role in multimodal integration.

Original languageEnglish (US)
Pages (from-to)1723-1737
Number of pages15
JournalJournal of cognitive neuroscience
Volume27
Issue number9
DOIs
StatePublished - 2015

ASJC Scopus subject areas

  • Cognitive Neuroscience

Fingerprint Dive into the research topics of 'Read my lips: Brain dynamics associated with audiovisual integration and deviance detection'. Together they form a unique fingerprint.

  • Cite this