BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations

Abstract

Pre-trained models are essential as feature extractors in modern machine learning systems in various domains. In this study, we hypothesize that representations effective for general audio tasks should provide multiple aspects of robust features of the input sound. For recognizing sounds regardless of perturbations such as varying pitch or timbre, features should be robust to these perturbations. For serving the diverse needs of tasks such as recognition of emotions or music genres, representations should provide multiple aspects of information, such as local and global features. To implement our principle, we propose a self-supervised learning method: Bootstrap Your Own Latent (BYOL) for Audio (BYOL-A, pronounced "viola"). BYOL-A pre-trains representations of the input sound invariant to audio data augmentations, which makes the learned representations robust to the perturbations of sounds. Whereas the BYOL-A encoder combines local and global features and calculates their statistics to make the representation provide multi-aspect information. As a result, the learned representations should provide robust and multi-aspect information to serve various needs of diverse tasks. We evaluated the general audio task performance of BYOL-A compared to previous state-of-the-art methods, and BYOL-A demonstrated generalizability with the best average result of 72.4% and the best VoxCeleb1 result of 57.6%. Extensive ablation experiments revealed that the BYOL-A encoder architecture contributes to most performance, and the final critical portion resorts to the BYOL framework and BYOL-A augmentations. Our code is available online at https://github.com/nttcslab/byol-a for future studies.

arXiv comment15 pages, 6 figures, and 15 tables. Under the review process

References in corpus (25)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• mixup: Beyond Empirical Risk Minimization
• Bootstrap your own latent: A new approach to self-supervised Learning
• VoxCeleb: a large-scale speaker identification dataset
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Towards Learning a Universal Non-Semantic Representation of Speech
• Receptive Field Regularization Techniques for Audio Classification and Tagging with Deep Convolutional Neural Networks
• SUPERB: Speech processing Universal PERformance Benchmark
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
• HEAR: Holistic Evaluation of Audio Representations
• Contrastive Learning of Musical Representations
• The NTT DCASE2020 Challenge Task 6 system: Automated Audio Captioning with Keywords and Sentence Length Estimation
• Multimodal Self-Supervised Learning of General Audio Representations
• Parsing Birdsong with Deep Audio Embeddings
• Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks

Cited by in corpus (7)

• Benchmarking Representations for Speech, Music, and Acoustic Events
• Parameter-Efficient Continual Fine-Tuning: A Survey
• Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning
• animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics
• SIGNL: A Label-Efficient Audio Deepfake Detection System via Spectral-Temporal Graph Non-Contrastive Learning
• From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation
• CardioPHON: Quality assessment and self-supervised pretraining for screening of cardiac function based on phonocardiogram recordings