An Empirical Evaluation of Educational Data Mining Techniques in a Dynamic VR Application

What makes an expert+ Beat Saber player? In the field of Educational Data Mining (EDM), there are various techniques for estimating latent skill mastery, such as Bayesian Knowledge Tracing (BKT) and Item Response Theory (IRT). While these techniques can estimate a student's skill level and even predict their performance, these EDM techniques have yet to be applied to dynamic, embodied motor tasks or immersive environments. In this work, we explore how these techniques may be used for VR learning applications and apply them to estimate latent skill mastery and task difficulty in a VR game similar to Beat Saber. We conducted a pilot study (n = 24) and a full study (n = 75) to collect empirical data with players of different skill levels in the VR game. While the EDM techniques lacked in accuracy, they provided opportunities such as helping identify flaws in the learning system design and the skill modeling. Through scrutinizing our methodology, we identify five challenges in applying these techniques to VR, provide insights into developing new robust assessment systems for VR learning environments and provide an agenda for future research.