Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?

Pooya Khorrami, Tom Le Paine, Thomas S. Huang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Despite being the appearance-based classifier of choice in recent years, relatively few works have examined how much convolutional neural networks (CNNs) can improve performance on accepted expression recognition benchmarks and, more importantly, examine what it is they actually learn. In this work, not only do we show that CNNs can achieve strong performance, but we also introduce an approach to decipher which portions of the face influence the CNN's predictions. First, we train a zero-bias CNN on facial expression data and achieve, to our knowledge, state-of-the-art performance on two expression recognition benchmarks: the extended Cohn-Kanade (CK+) dataset and the Toronto Face Dataset (TFD). We then qualitatively analyze the network by visualizing the spatial patterns that maximally excite different neurons in the convolutional layers and show how they resemble Facial Action Units (FAUs). Finally, we use the FAU labels provided in the CK+ dataset to verify that the FAUs observed in our filter visualizations indeed align with the subject's facial movements.

Original languageEnglish (US)
Title of host publicationProceedings - 2015 IEEE International Conference on Computer Vision Workshops, ICCVW 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages19-27
Number of pages9
ISBN (Electronic)9781467383905
DOIs
StatePublished - Feb 11 2015
Event15th IEEE International Conference on Computer Vision Workshops, ICCVW 2015 - Santiago, Chile
Duration: Dec 11 2015Dec 18 2015

Publication series

NameProceedings of the IEEE International Conference on Computer Vision
Volume2015-February
ISSN (Print)1550-5499

Other

Other15th IEEE International Conference on Computer Vision Workshops, ICCVW 2015
CountryChile
CitySantiago
Period12/11/1512/18/15

Keywords

  • Benchmark testing
  • Biological neural networks
  • Databases
  • Emotion recognition
  • Face
  • Face recognition
  • Training

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition

Fingerprint Dive into the research topics of 'Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?'. Together they form a unique fingerprint.

Cite this