From Notes to Keys: A VR Learning Environment for Sheet Music Interpretation

Learning to interpret sheet music and play musical instruments (piano) remains a significant challenge for beginners, often requiring extensive practice and guidance. Existing platforms lack real-time feedback and seamless sheet music interpretation, creating inefficiencies that require a system for accurate note recognition, intuitive guidance, and reduced cognitive load. To address this, we propose a novel system that integrates optical music recognition (OMR) and virtual reality (VR) to create an immersive piano learning environment. The proposed approach improves the detection of musical notes by making it scale and rotation invariant. The detected notes are converted into the corresponding piano keys and sequential instructions. These instructions are then visualized in a VR environment in Meta Quest 3, where a virtual piano highlights keys dynamically to guide the user. The experimental results demonstrate high accuracy in note detection and significant improvements in the learning curve for beginners, reducing cognitive load, and bridging the gap between sheet music interpretation and piano playing. This work highlights the potential of combining document image analysis and VR technologies to revolutionize music education, as well as other related fields, offering a scalable and accessible solution.