Multi-Format Contrastive Learning of Audio Representations
arXiv:2103.06508
Abstract
Recent advances suggest the advantage of multi-modal training in comparison with single-modal methods. In contrast to this view, in our work we find that similar gain can be obtained from training with different formats of a single modality. In particular, we investigate the use of the contrastive learning framework to learn audio representations by maximizing the agreement between the raw audio and its spectral representation. We find a significant gain using this multi-format strategy against the single-format counterparts. Moreover, on the downstream AudioSet and ESC-50 classification task, our audio-only approach achieves new state-of-the-art results with a mean average precision of 0.376 and an accuracy of 90.5%, respectively.
References in corpus (9)
- A Simple Framework for Contrastive Learning of Visual Representations
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Learning deep representations by mutual information estimation and maximization
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Big Self-Supervised Models are Strong Semi-Supervised Learners
- Libri-Light: A Benchmark for ASR with Limited or No Supervision
- Self-Supervised Learning by Cross-Modal Audio-Video Clustering
- Self-Supervised MultiModal Versatile Networks
- Learning Robust and Multilingual Speech Representations
Cited by in corpus (9)
- BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
- OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
- Multimodal Self-Supervised Learning of General Audio Representations
- Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning
- Contrastive Neural Processes for Self-Supervised Learning
- Towards Learning Universal Audio Representations
- Learning source-aware representations of music in a discrete latent space
- An empirical study of weakly supervised audio tagging embeddings for general audio representations
- The Impact of Spatiotemporal Augmentations on Self-Supervised Audiovisual Representation Learning