PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Music

Abstract

Definitive embeddings remain a fundamental challenge of computational musicology for symbolic music in deep learning today. Analogous to natural language, music can be modeled as a sequence of tokens. This motivates the majority of existing solutions to explore the utilization of word embedding models to build music embeddings. However, music differs from natural languages in two key aspects: (1) musical token is multi-faceted -- it comprises of pitch, rhythm and dynamics information; and (2) musical context is two-dimensional -- each musical token is dependent on both melodic and harmonic contexts. In this work, we provide a comprehensive solution by proposing a novel framework named PiRhDy that integrates pitch, rhythm, and dynamics information seamlessly. PiRhDy adopts a hierarchical strategy which can be decomposed into two steps: (1) token (i.e., note event) modeling, which separately represents pitch, rhythm, and dynamics and integrates them into a single token embedding; and (2) context modeling, which utilizes melodic and harmonic knowledge to train the token embedding. A thorough study was made on each component and sub-strategy of PiRhDy. We further validate our embeddings in three downstream tasks -- melody completion, accompaniment suggestion, and genre classification. Results indicate a significant advancement of the neural approach towards symbolic music as well as PiRhDy's potential as a pretrained tool for a broad range of symbolic music applications.

arXiv commentACM Multimedia 2020 -- best paper

References in corpus (3)

• Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems
• DopeLearning: A Computational Approach to Rap Lyrics Generation
• On large-scale genre classification in symbolically encoded music by automatic identification of repeating patterns

Cited by in corpus (4)

• Formalizing Multimedia Recommendation through Multimodal Deep Learning
• Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey
• Learning Music-Dance Representations through Explicit-Implicit Rhythm Synchronization
• DiVa: An Iterative Framework to Harvest More Diverse and Valid Labels from User Comments for Music