Tracking and Recognition of Fist Move on Musical Excerpts at the Piano

Tracking and Recognition of Fist Move on Musical Excerpts at the Piano

Abstract

This works presents a method for teaching assistance in exercises of piano playing with hand tracking and gesture recognition. A tutor defines exercises to be performed and the system detects and tracks the hands of student and checks the correct perform of exercises. To hand detection and tracking are used coloured markers on the back of hands, captured by a regular webcam and for gesture recognition is used Multidimensional Dynamic Time Warping (MDTW), a n-dimensional version of Dynamic Time Warping (DTW). To verify the performance of the method, we have made videos of exercises, which were filtered for some adjusts and noise reduction. Results shows that the method has been capable to detect and recognize gestures successfully. Although the system needs some improvements, first impression about the system has been satisfactory. Thus, we hope, until the end of this work, implement a real-time version of the method for build a tutor system for piano classes.

Publication
In Proceedings of 11th Brazilian Workshop on Virtual and Augmented Reality (WRVA)
Date
Links
PDF

Text in Portuguese, with English title and abstract.