Robust multi-modal cues for dyadic human interaction recognition

Rim Trabelsi; Jagannadan Varadarajan; Yong Pei; Le Zhang; Issam Jabri; Ammar Bouallegue; Pierre Moulin

doi:10.1145/3132515.3132517

Robust multi-modal cues for dyadic human interaction recognition

Rim Trabelsi, Jagannadan Varadarajan, Yong Pei, Le Zhang, Issam Jabri, Ammar Bouallegue, Pierre Moulin

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Activity analysis methods usually tend to focus on elementary human actions but ignore to analyze complex scenarios. In this paper, we focus particularly on classifying interactions between two persons in a supervised fashion. We propose a robust multi-modal proxemic descriptor based on 3D joint locations, depth and color videos. The proposed descriptor incorporates inter-person and intraperson joint distances calculated from 3D skeleton data and multiframe dense optical flow features obtained from the application of temporal Convolutional neural networks (CNN) on depth and color images. The descriptors from the three modalities are derived from sparse key-frames surrounding high activity content and fused using a linear SVM classifier. Through experiments on two publicly available RGB-D interaction datasets, we show that our method can efficiently classify complex interactions using only short video snippet, outperforming existing state-of-the-art results.

Original language	English (US)
Title of host publication	MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017
Publisher	Association for Computing Machinery
Pages	47-53
Number of pages	7
ISBN (Electronic)	9781450355094
DOIs	https://doi.org/10.1145/3132515.3132517
State	Published - Oct 27 2017
Event	1st ACM MM Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, MUSA2 2017 - Mountain View, United States Duration: Oct 27 2017 → …

Publication series

Name	MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017

Other

Other	1st ACM MM Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, MUSA2 2017
Country/Territory	United States
City	Mountain View
Period	10/27/17 → …

Keywords

CNN features
Interaction recognition
Multi-modal features
RGB-D
Skeleton data

ASJC Scopus subject areas

Human-Computer Interaction
Computer Graphics and Computer-Aided Design

Online availability

10.1145/3132515.3132517

Library availability

Discover UIUC Full Text

Cite this

Trabelsi, R., Varadarajan, J., Pei, Y., Zhang, L., Jabri, I., Bouallegue, A., & Moulin, P. (2017). Robust multi-modal cues for dyadic human interaction recognition. In MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017 (pp. 47-53). (MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017). Association for Computing Machinery. https://doi.org/10.1145/3132515.3132517

Robust multi-modal cues for dyadic human interaction recognition. / Trabelsi, Rim; Varadarajan, Jagannadan; Pei, Yong et al.
MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017. Association for Computing Machinery, 2017. p. 47-53 (MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Trabelsi, R, Varadarajan, J, Pei, Y, Zhang, L, Jabri, I, Bouallegue, A & Moulin, P 2017, Robust multi-modal cues for dyadic human interaction recognition. in MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017. MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017, Association for Computing Machinery, pp. 47-53, 1st ACM MM Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, MUSA2 2017, Mountain View, United States, 10/27/17. https://doi.org/10.1145/3132515.3132517

Trabelsi R, Varadarajan J, Pei Y, Zhang L, Jabri I, Bouallegue A et al. Robust multi-modal cues for dyadic human interaction recognition. In MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017. Association for Computing Machinery. 2017. p. 47-53. (MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017). doi: 10.1145/3132515.3132517

Trabelsi, Rim ; Varadarajan, Jagannadan ; Pei, Yong et al. / Robust multi-modal cues for dyadic human interaction recognition. MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017. Association for Computing Machinery, 2017. pp. 47-53 (MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017).

@inproceedings{a8b0a553d6e94792b03fe72c8cf845fb,

title = "Robust multi-modal cues for dyadic human interaction recognition",

abstract = "Activity analysis methods usually tend to focus on elementary human actions but ignore to analyze complex scenarios. In this paper, we focus particularly on classifying interactions between two persons in a supervised fashion. We propose a robust multi-modal proxemic descriptor based on 3D joint locations, depth and color videos. The proposed descriptor incorporates inter-person and intraperson joint distances calculated from 3D skeleton data and multiframe dense optical flow features obtained from the application of temporal Convolutional neural networks (CNN) on depth and color images. The descriptors from the three modalities are derived from sparse key-frames surrounding high activity content and fused using a linear SVM classifier. Through experiments on two publicly available RGB-D interaction datasets, we show that our method can efficiently classify complex interactions using only short video snippet, outperforming existing state-of-the-art results.",

keywords = "CNN features, Interaction recognition, Multi-modal features, RGB-D, Skeleton data",

author = "Rim Trabelsi and Jagannadan Varadarajan and Yong Pei and Le Zhang and Issam Jabri and Ammar Bouallegue and Pierre Moulin",

note = "Publisher Copyright: {\textcopyright} 2017 Copyright held by the owner/author(s).; 1st ACM MM Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, MUSA2 2017 ; Conference date: 27-10-2017",

year = "2017",

month = oct,

day = "27",

doi = "10.1145/3132515.3132517",

language = "English (US)",

series = "MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017",

publisher = "Association for Computing Machinery",

pages = "47--53",

booktitle = "MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017",

address = "United States",

}

TY - GEN

T1 - Robust multi-modal cues for dyadic human interaction recognition

AU - Trabelsi, Rim

AU - Varadarajan, Jagannadan

AU - Pei, Yong

AU - Zhang, Le

AU - Jabri, Issam

AU - Bouallegue, Ammar

AU - Moulin, Pierre

PY - 2017/10/27

Y1 - 2017/10/27

N2 - Activity analysis methods usually tend to focus on elementary human actions but ignore to analyze complex scenarios. In this paper, we focus particularly on classifying interactions between two persons in a supervised fashion. We propose a robust multi-modal proxemic descriptor based on 3D joint locations, depth and color videos. The proposed descriptor incorporates inter-person and intraperson joint distances calculated from 3D skeleton data and multiframe dense optical flow features obtained from the application of temporal Convolutional neural networks (CNN) on depth and color images. The descriptors from the three modalities are derived from sparse key-frames surrounding high activity content and fused using a linear SVM classifier. Through experiments on two publicly available RGB-D interaction datasets, we show that our method can efficiently classify complex interactions using only short video snippet, outperforming existing state-of-the-art results.

AB - Activity analysis methods usually tend to focus on elementary human actions but ignore to analyze complex scenarios. In this paper, we focus particularly on classifying interactions between two persons in a supervised fashion. We propose a robust multi-modal proxemic descriptor based on 3D joint locations, depth and color videos. The proposed descriptor incorporates inter-person and intraperson joint distances calculated from 3D skeleton data and multiframe dense optical flow features obtained from the application of temporal Convolutional neural networks (CNN) on depth and color images. The descriptors from the three modalities are derived from sparse key-frames surrounding high activity content and fused using a linear SVM classifier. Through experiments on two publicly available RGB-D interaction datasets, we show that our method can efficiently classify complex interactions using only short video snippet, outperforming existing state-of-the-art results.

KW - CNN features

KW - Interaction recognition

KW - Multi-modal features

KW - RGB-D

KW - Skeleton data

UR - http://www.scopus.com/inward/record.url?scp=85035747317&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85035747317&partnerID=8YFLogxK

U2 - 10.1145/3132515.3132517

DO - 10.1145/3132515.3132517

M3 - Conference contribution

AN - SCOPUS:85035747317

T3 - MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017

SP - 47

EP - 53

BT - MUSA2 2017 - Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, co-located with MM 2017

PB - Association for Computing Machinery

T2 - 1st ACM MM Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes, MUSA2 2017

Y2 - 27 October 2017

ER -

Robust multi-modal cues for dyadic human interaction recognition

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this