Enriched Music Representations with Multiple Cross-modal Contrastive Learning

Abstract

Modeling various aspects that make a music piece unique is a challenging task, requiring the combination of multiple sources of information. Deep learning is commonly used to obtain representations using various sources of information, such as the audio, interactions between users and songs, or associated genre metadata. Recently, contrastive learning has led to representations that generalize better compared to traditional supervised methods. In this paper, we present a novel approach that combines multiple types of information related to music using cross-modal contrastive learning, allowing us to learn an audio feature from heterogeneous data simultaneously. We align the latent representations obtained from playlists-track interactions, genre metadata, and the tracks' audio, by maximizing the agreement between these modality representations using a contrastive loss. We evaluate our approach in three tasks, namely, genre classification, playlist continuation and automatic tagging. We compare the performances with a baseline audio-based CNN trained to predict these modalities. We also study the importance of including multiple sources of information when training our embedding model. The results suggest that the proposed method outperforms the baseline in all the three downstream tasks and achieves comparable performance to the state-of-the-art.

Accepted for publication to IEEE Signal Processing Letters

References in corpus (14)

• Efficient Estimation of Word Representations in Vector Space
• A Simple Framework for Contrastive Learning of Visual Representations
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Representation Learning with Contrastive Predictive Coding
• Contrastive Representation Learning: A Framework and Review
• Supervised Contrastive Learning
• musicnn: Pre-trained convolutional neural networks for music audio tagging
• Evaluation of CNN-based Automatic Music Tagging Models
• Classification is a Strong Baseline for Deep Metric Learning
• Zero-shot Learning for Audio-based Music Classification and Tagging
• Metric Learning vs Classification for Disentangled Music Representation Learning
• COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations
• Multimodal Metric Learning for Tag-based Music Retrieval
• Mood Classification Using Listening Data

Cited by in corpus (4)

• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• Codified audio language modeling learns useful representations for music information retrieval
• Bridging the Gap Between Semantic and User Preference Spaces for Multi-modal Music Representation Learning