Towards Learning Universal Audio Representations

Abstract

The ability to learn universal audio representations that can solve diverse speech, music, and environment tasks can spur many applications that require general sound content understanding. In this work, we introduce a holistic audio representation evaluation suite (HARES) spanning 12 downstream tasks across audio domains and provide a thorough empirical study of recent sound representation learning systems on that benchmark. We discover that previous sound event classification or speech models do not generalize outside of their domains. We observe that more robust audio representations can be learned with the SimCLR objective; however, the model's transferability depends heavily on the model architecture. We find the Slowfast architecture is good at learning rich representations required by different domains, but its performance is affected by the normalization scheme. Based on these findings, we propose a novel normalizer-free Slowfast NFNet and achieve state-of-the-art performance across all domains.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Instance Normalization: The Missing Ingredient for Fast Stylization
• VoxCeleb: a large-scale speaker identification dataset
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge
• Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
• High-Performance Large-Scale Image Recognition Without Normalization
• Towards Learning a Universal Non-Semantic Representation of Speech
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• End-to-end learning for music audio tagging at scale
• PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
• SUPERB: Speech processing Universal PERformance Benchmark
• AST: Audio Spectrogram Transformer
• Contrastive Learning of Musical Representations
• Multi-Format Contrastive Learning of Audio Representations
• Characterizing signal propagation to close the performance gap in unnormalized ResNets
• Multimodal Self-Supervised Learning of General Audio Representations