Towards Learning a Universal Non-Semantic Representation of Speech

Abstract

The ultimate goal of transfer learning is to reduce labeled data requirements by exploiting a pre-existing embedding model trained for different datasets or tasks. The visual and language communities have established benchmarks to compare embeddings, but the speech community has yet to do so. This paper proposes a benchmark for comparing speech representations on non-semantic tasks, and proposes a representation based on an unsupervised triplet-loss objective. The proposed representation outperforms other representations on the benchmark, and even exceeds state-of-the-art performance on a number of transfer learning tasks. The embedding is trained on a publicly available dataset, and it is tested on a variety of low-resource downstream tasks, including personalization tasks and medical domain. The benchmark, models, and evaluation code are publicly released.

References in corpus (16)

• Scikit-learn: Machine Learning in Python
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Representation Learning with Contrastive Predictive Coding
• How transferable are features in deep neural networks?
• VoxCeleb: a large-scale speaker identification dataset
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Look, Listen and Learn
• Multiclass Language Identification using Deep Learning on Spectral Images of Audio Signals
• On the importance of normative data in speech-based assessment
• auDeep: Unsupervised Learning of Representations from Audio with Deep Recurrent Neural Networks
• Attention-Augmented End-to-End Multi-Task Learning for Emotion Prediction from Speech
• Unsupervised Learning of Semantic Audio Representations
• Deep Audio Prior

Cited by in corpus (25)

• Self-Supervised Speech Representation Learning: A Review
• PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations
• SUPERB: Speech processing Universal PERformance Benchmark
• Learning Efficient Representations for Keyword Spotting with Triplet Loss
• Detecting Dementia from Speech and Transcripts using Transformers
• TRILLsson: Distilled Universal Paralinguistic Speech Representations
• Universal Paralinguistic Speech Representations Using Self-Supervised Conformers
• Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP
• ATST: Audio Representation Learning with Teacher-Student Transformer
• Multimodal Self-Supervised Learning of General Audio Representations
• FRILL: A Non-Semantic Speech Embedding for Mobile Devices
• DECAR: Deep Clustering for learning general-purpose Audio Representations
• Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings
• A Cough-based deep learning framework for detecting COVID-19
• Transfer Learning with Jukebox for Music Source Separation
• Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices
• Sound-Dr: Reliable Sound Dataset and Baseline Artificial Intelligence System for Respiratory Illnesses
• A Multi-modal Deep Learning Model for Video Thumbnail Selection
• Relationship between auditory and semantic entrainment using Deep Neural Networks (DNN)
• Towards Learning Universal Audio Representations
• Temporal Convolution Networks with Positional Encoding for Evoked Expression Estimation
• Active Learning of Non-semantic Speech Tasks with Pretrained Models
• Scenario Aware Speech Recognition: Advancements for Apollo Fearless Steps & CHiME-4 Corpora