Learning Disentangled Representations for Timber and Pitch in Music Audio

Abstract

Timbre and pitch are the two main perceptual properties of musical sounds. Depending on the target applications, we sometimes prefer to focus on one of them, while reducing the effect of the other. Researchers have managed to hand-craft such timbre-invariant or pitch-invariant features using domain knowledge and signal processing techniques, but it remains difficult to disentangle them in the resulting feature representations. Drawing upon state-of-the-art techniques in representation learning, we propose in this paper two deep convolutional neural network models for learning disentangled representation of musical timbre and pitch. Both models use encoders/decoders and adversarial training to learn music representations, but the second model additionally uses skip connections to deal with the pitch information. As music is an art of time, the two models are supervised by frame-level instrument and pitch labels using a new dataset collected from MuseScore. We compare the result of the two disentangling models with a new evaluation protocol called "timbre crossover", which leads to interesting applications in audio-domain music editing. Via various objective evaluations, we show that the second model can better change the instrumentation of a multi-instrument music piece without much affecting the pitch structure. By disentangling timbre and pitch, we envision that the model can contribute to generating more realistic music audio as well.

References in corpus (7)

• WaveNet: A Generative Model for Raw Audio
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation
• Discovering Hidden Factors of Variation in Deep Networks
• MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer
• Learning a Latent Space of Multitrack Measures
• Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation

Cited by in corpus (5)

• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling
• Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders
• Musical Composition Style Transfer via Disentangled Timbre Representations
• Visual Attention for Musical Instrument Recognition