A mixed reality system for teaching STEM content using embodied learning and whole-body metaphors

This paper describes the development of an MR environment that can be used in teaching STEM (Science, Technology, Engineering, and Mathematics) topics. Specifically we seek to create a space for facilitating whole-body metaphors where learners use the physical movement and positioning of their entire bodies to enact their understanding of complex concepts. A rigorous technical approach comprised of virtual elements, real users, spatial audio, and an integrated sensor network is presented that fulfills the requirements of an embodied learning environment. An algorithm that uses homography-based multi-projector blending is used to create a large, seamless projection on the floor that affords a human-scale interaction environment. To further improve the immersive quality, projectors are strategically overlapped to minimize user shadows on the projected surface. A hybrid sensor solution using a Kinect and a laser scanner is developed that tracks users' physical movements and extracts relevant game parameters such as position and velocity. Requiring no pre-training or props, this tracking setup is adaptable and shows high performance over a wide range of users, from children to adults. An exhibit employing this MR system was field-tested at the Museum of Science and Industry in Tampa, FL.