SKY-Piano: A Multimodal Piano Performance Dataset
arXiv:2607.27296
The paper introduces SKY-Piano, a multimodal dataset of piano performances that includes audio, MIDI, multi-view video, hand and body motion capture, and MusicXML scores from professional and amateur pianists, along with tools for browsing, fingering annotation, and a MIDI-to-motion generation example.
Abstract
Music information retrieval research on piano performance increasingly involves diverse modalities of data and annotations beyond audio and MIDI. We present SKY-Piano, a multimodal piano performance dataset that includes 11 hours of performance recordings of motion, multi-view video, audio, MIDI from 7 professional and 12 amateur pianists along with MusicXML scores. The performance pieces were selected considering playing technique, difficulty, and performer expertise on a shared core repertoire. The motion data include both hand and body motion, released in both flagged form, where samples lost to marker occlusion are marked as unreliable, and imputed form, where those gaps are reconstructed, together with Visual3D body-segment kinematics and other time-synchronized modalities. To easily browse different modalities of data at a glance, we provide an interactive web browser. In addition, we developed a fingering annotation model and tool for deriving pseudo fingering annotations from the MIDI and motion data. Lastly, we present MIDI-to-motion generation through a fine-tuning experiment as a use case of the dataset.
Accepted to the 27th International Society for Music Information Retrieval Conference (ISMIR 2026), Abu Dhabi, UAE. Project page: https://joonhyungbae.github.io/skypiano/