Multi-modal social interaction recognition using view-invariant features

Rim Trabelsi, Jagannadan Varadarajan, Yong Pei, Le Zhang, Issam Jabri, Ammar Bouallegue, Pierre Moulin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper addresses the issue of analyzing social interactions between humans in videos. We focus on recognizing dyadic human interactions through multi-modal data, specifically, depth, color and skeleton sequences. Firstly, we introduce a new person-centric proxemic descriptor, named PROF, extracted from skeleton data able to incorporate intrinsic and extrinsic distances between two interacting persons in a view-variant scheme. Then, a novel key frame selection approach is introduced to identify salient instants of the interaction sequence based on the joint energy. From RGBD videos, more holistic CNN features are extracted by applying an adaptive pre-trained CNNs on optical flow frames. Features from three modalities are combined then classified using linear SVM. Finally, extensive experiments have been carried on two multi-modal and multi-view interactions datasets prove the robustness of the introduced approach comparing to state-of-the-art methods.

Original languageEnglish (US)
Title of host publicationISIAA 2017 - Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents, Co-located with ICMI 2017
EditorsFabrice Lefevre, Thierry Chaminade, Noel Ngyuen, Magalie Ochs
PublisherAssociation for Computing Machinery, Inc
Pages47-48
Number of pages2
ISBN (Electronic)9781450355582
DOIs
StatePublished - Nov 13 2017
Event1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents, ISIAA 2017 - Glasgow, United Kingdom
Duration: Nov 13 2017 → …

Publication series

NameISIAA 2017 - Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents, Co-located with ICMI 2017

Other

Other1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents, ISIAA 2017
CountryUnited Kingdom
CityGlasgow
Period11/13/17 → …

Keywords

  • Multi-modal data
  • Multi-view
  • Social interaction

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Human-Computer Interaction
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Multi-modal social interaction recognition using view-invariant features'. Together they form a unique fingerprint.

Cite this