Contrastive Learning of Musical Representations

Abstract

While deep learning has enabled great advances in many areas of music, labeled music datasets remain especially hard, expensive, and time-consuming to create. In this work, we introduce SimCLR to the music domain and contribute a large chain of audio data augmentations to form a simple framework for self-supervised, contrastive learning of musical representations: CLMR. This approach works on raw time-domain music data and requires no labels to learn useful representations. We evaluate CLMR in the downstream task of music classification on the MagnaTagATune and Million Song datasets and present an ablation study to test which of our music-related innovations over SimCLR are most effective. A linear classifier trained on the proposed representations achieves a higher average precision than supervised models on the MagnaTagATune dataset, and performs comparably on the Million Song dataset. Moreover, we show that CLMR's representations are transferable using out-of-domain datasets, indicating that our method has strong generalisability in music classification. Lastly, we show that the proposed method allows data-efficient learning on smaller labeled datasets: we achieve an average precision of 33.1% despite using only 259 labeled songs in the MagnaTagATune dataset (1% of the full dataset) during linear evaluation. To foster reproducibility and future research on self-supervised learning in music, we publicly release the pre-trained models and the source code of all experiments of this paper.

arXiv comment15 pages, 8 figures. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR, 2021

References in corpus (10)

• A Simple Framework for Contrastive Learning of Visual Representations
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Bootstrap your own latent: A new approach to self-supervised Learning
• Learning deep representations by mutual information estimation and maximization
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• A Fully Convolutional Deep Auditory Model for Musical Chord Recognition

Cited by in corpus (7)

• BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations
• Codified audio language modeling learns useful representations for music information retrieval
• Parsing Birdsong with Deep Audio Embeddings
• SpecTNT: a Time-Frequency Transformer for Music Audio
• Learning Music Sequence Representation from Text Supervision
• Towards Learning Universal Audio Representations
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations