Enriching Multimodal Data: A Temporal Approach to Contextualize Joint Attention in Collaborative Problem-Solving

Yiqiu Zhou, Jina Kang

Research output: Contribution to journalArticlepeer-review


Collaboration is a complex, multidimensional process; however, details of how multimodal features intersect and mediate group interactions have not been fully unpacked. Characterizing and analyzing the temporal patterns based on multimodal features is a challenging yet important work to advance our understanding of computer-supported collaborative learning (CSCL). This paper highlights the affordances, as well as the limitations, of different temporal approaches in terms of analyzing multimodal data. To tackle the remaining challenges, we present an empirical example of multimodal temporal analysis that leverages multi-level vector autoregression (mlVAR) to identify temporal patterns of the collaborative problem-solving (CPS) process in an immersive astronomy simulation. We extend previous research on joint attention with a particular focus on the added value from a multimodal, temporal account of the CPS process. We incorporate verbal discussion to contextualize joint attention, examine the sequential and contemporaneous associations between them, and identify significant differences in temporal patterns between low-and high-achieving groups. Our paper does the following: 1) creates interpretable multimodal group interaction patterns, 2) advances understanding of CPS through examination of verbal and non-verbal interactions, and 3) demonstrates the added value of a complete account of temporality including both duration and sequential order.

Original languageEnglish (US)
Pages (from-to)87-101
Number of pages15
JournalJournal of Learning Analytics
Issue number3
StatePublished - Dec 22 2023


  • CSCL
  • Collaborative problem-solving
  • collaboration analytics
  • multimodal learning analytics
  • temporal analysis
  • temporality

ASJC Scopus subject areas

  • Education
  • Computer Science Applications


Dive into the research topics of 'Enriching Multimodal Data: A Temporal Approach to Contextualize Joint Attention in Collaborative Problem-Solving'. Together they form a unique fingerprint.

Cite this