Pipelining localized semantic features for fine-grained action recognition

Yang Zhou, Bingbing Ni, Shuicheng Yan, Pierre Moulin, Qi Tian

Research output: Chapter in Book/Report/Conference proceedingConference contribution


In fine-grained action (object manipulation) recognition, it is important to encode object semantic (contextual) information, i.e., which object is being manipulated and how it is being operated. However, previous methods for action recognition often represent the semantic information in a global and coarse way and therefore cannot cope with fine-grained actions. In this work, we propose a representation and classification pipeline which seamlessly incorporates localized semantic information into every processing step for fine-grained action recognition. In the feature extraction stage, we explore the geometric information between local motion features and the surrounding objects. In the feature encoding stage, we develop a semantic-grouped locality-constrained linear coding (SG-LLC) method that captures the joint distributions between motion and object-in-use information. Finally, we propose a semantic-aware multiple kernel learning framework (SA-MKL) by utilizing the empirical joint distribution between action and object type for more discriminative action classification. Extensive experiments are performed on the large-scale and difficult fine-grained MPII cooking action dataset. The results show that by effectively accumulating localized semantic information into the action representation and classification pipeline, we significantly improve the fine-grained action classification performance over the existing methods.

Original languageEnglish (US)
Title of host publicationComputer Vision, ECCV 2014 - 13th European Conference, Proceedings
Number of pages16
EditionPART 4
ISBN (Print)9783319105925
StatePublished - 2014
Event13th European Conference on Computer Vision, ECCV 2014 - Zurich, Switzerland
Duration: Sep 6 2014Sep 12 2014

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
NumberPART 4
Volume8692 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Other13th European Conference on Computer Vision, ECCV 2014

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'Pipelining localized semantic features for fine-grained action recognition'. Together they form a unique fingerprint.

Cite this