Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior

Iris I.A. Groen, Michelle R. Greene, Christopher Baldassano, Li Fei-Fei, Diane M Beck, Chris I. Baker

Research output: Contribution to journalArticle

Abstract

Inherent correlations between visual and semantic features in real-world scenes make it difficult to determine how different scene properties contribute to neural representations. Here, we assessed the contributions of multiple properties to scene representation by partitioning the variance explained in human behavioral and brain measurements by three feature models whose inter-correlations were minimized a priori through stimulus preselection. Behavioral assessments of scene similarity reflected unique contributions from a functional feature model indicating potential actions in scenes as well as high-level visual features from a deep neural network (DNN). In contrast, similarity of cortical responses in scene-selective areas was uniquely explained by mid-and high-level DNN features only, while an object label model did not contribute uniquely to either domain. The striking dissociation between functional and DNN features in their contribution to behavioral and brain representations of scenes indicates that scene-selective cortex represents only a subset of behaviorally relevant scene information.

Original languageEnglish (US)
Article numbere32962
JournaleLife
Volume7
DOIs
StatePublished - Mar 7 2018

Fingerprint

Brain
Semantics
Action Potentials
Labels
Deep neural networks

ASJC Scopus subject areas

  • Neuroscience(all)
  • Biochemistry, Genetics and Molecular Biology(all)
  • Immunology and Microbiology(all)

Cite this

Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior. / Groen, Iris I.A.; Greene, Michelle R.; Baldassano, Christopher; Fei-Fei, Li; Beck, Diane M; Baker, Chris I.

In: eLife, Vol. 7, e32962, 07.03.2018.

Research output: Contribution to journalArticle

Groen, Iris I.A. ; Greene, Michelle R. ; Baldassano, Christopher ; Fei-Fei, Li ; Beck, Diane M ; Baker, Chris I. / Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior. In: eLife. 2018 ; Vol. 7.
@article{215a9e457fd44d519c57fe751899916b,
title = "Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior",
abstract = "Inherent correlations between visual and semantic features in real-world scenes make it difficult to determine how different scene properties contribute to neural representations. Here, we assessed the contributions of multiple properties to scene representation by partitioning the variance explained in human behavioral and brain measurements by three feature models whose inter-correlations were minimized a priori through stimulus preselection. Behavioral assessments of scene similarity reflected unique contributions from a functional feature model indicating potential actions in scenes as well as high-level visual features from a deep neural network (DNN). In contrast, similarity of cortical responses in scene-selective areas was uniquely explained by mid-and high-level DNN features only, while an object label model did not contribute uniquely to either domain. The striking dissociation between functional and DNN features in their contribution to behavioral and brain representations of scenes indicates that scene-selective cortex represents only a subset of behaviorally relevant scene information.",
author = "Groen, {Iris I.A.} and Greene, {Michelle R.} and Christopher Baldassano and Li Fei-Fei and Beck, {Diane M} and Baker, {Chris I.}",
year = "2018",
month = "3",
day = "7",
doi = "10.7554/eLife.32962",
language = "English (US)",
volume = "7",
journal = "eLife",
issn = "2050-084X",
publisher = "eLife Sciences Publications",

}

TY - JOUR

T1 - Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior

AU - Groen, Iris I.A.

AU - Greene, Michelle R.

AU - Baldassano, Christopher

AU - Fei-Fei, Li

AU - Beck, Diane M

AU - Baker, Chris I.

PY - 2018/3/7

Y1 - 2018/3/7

N2 - Inherent correlations between visual and semantic features in real-world scenes make it difficult to determine how different scene properties contribute to neural representations. Here, we assessed the contributions of multiple properties to scene representation by partitioning the variance explained in human behavioral and brain measurements by three feature models whose inter-correlations were minimized a priori through stimulus preselection. Behavioral assessments of scene similarity reflected unique contributions from a functional feature model indicating potential actions in scenes as well as high-level visual features from a deep neural network (DNN). In contrast, similarity of cortical responses in scene-selective areas was uniquely explained by mid-and high-level DNN features only, while an object label model did not contribute uniquely to either domain. The striking dissociation between functional and DNN features in their contribution to behavioral and brain representations of scenes indicates that scene-selective cortex represents only a subset of behaviorally relevant scene information.

AB - Inherent correlations between visual and semantic features in real-world scenes make it difficult to determine how different scene properties contribute to neural representations. Here, we assessed the contributions of multiple properties to scene representation by partitioning the variance explained in human behavioral and brain measurements by three feature models whose inter-correlations were minimized a priori through stimulus preselection. Behavioral assessments of scene similarity reflected unique contributions from a functional feature model indicating potential actions in scenes as well as high-level visual features from a deep neural network (DNN). In contrast, similarity of cortical responses in scene-selective areas was uniquely explained by mid-and high-level DNN features only, while an object label model did not contribute uniquely to either domain. The striking dissociation between functional and DNN features in their contribution to behavioral and brain representations of scenes indicates that scene-selective cortex represents only a subset of behaviorally relevant scene information.

UR - http://www.scopus.com/inward/record.url?scp=85045639406&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85045639406&partnerID=8YFLogxK

U2 - 10.7554/eLife.32962

DO - 10.7554/eLife.32962

M3 - Article

C2 - 29513219

AN - SCOPUS:85045639406

VL - 7

JO - eLife

JF - eLife

SN - 2050-084X

M1 - e32962

ER -