Gesture Patterns and Learning in an Embodied XR Science Simu

Recent research has emphasized the importance of leveraging embodied interactions for learning critical Stem concepts. Elastic³s—an embodied environment for learning about cross-cutting concepts (i.e., non-linear growth)—allows learners to interact with different science simulations through whole-body gestures. Technological advances in gesture recognition can track and respond to students’ gestures, however, there has been little investigation into how the gestures performed in these environments relate to subsequent learning. The need for sequential pattern recognition methods is critical in embodied learning if we are to understand how gestural interaction with a simulation facilitates learning. Using data collected via Microsoft Kinect V2 from twelve college students, we applied multivariate Dynamic Time Warping for clustering to identify gestural patterns in Elastic³s as evidence for embodied learning processes. Our findings showed that identified trends of simulation use were indicative of students’ struggles to understand the underlying ideas or use of the system and were associated with learning performance. These indicators can potentially be used to leverage real time, in-simulation assistance and promote a more adaptive learning experience via embodied simulations.