TRILLsson: Distilled Universal Paralinguistic Speech Representations

Abstract

Recent advances in self-supervision have dramatically improved the quality of speech representations. However, deployment of state-of-the-art embedding models on devices has been restricted due to their limited public availability and large resource footprint. Our work addresses these issues by publicly releasing a collection of paralinguistic speech models that are small and near state-of-the-art performance. Our approach is based on knowledge distillation, and our models are distilled on public data only. We explore different architectures and thoroughly evaluate our models on the Non-Semantic Speech (NOSS) benchmark. Our largest distilled model is less than 15% the size of the original model (314MB vs 2.2GB), achieves over 96% the accuracy on 6 of 7 tasks, and is trained on 6.5% the data. The smallest model is 1% in size (22MB) and achieves over 90% the accuracy on 6 of 7 tasks. Our models outperform the open source Wav2Vec 2.0 model on 6 of 7 tasks, and our smallest model outperforms the open source Wav2Vec 2.0 on both emotion recognition tasks despite being 7% the size.

Submitted to Interspeech 2022

References in corpus (4)

• Distilling the Knowledge in a Neural Network
• EfficientNetV2: Smaller Models and Faster Training
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Universal Paralinguistic Speech Representations Using Self-Supervised Conformers

Cited by in corpus (6)

• Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP
• Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings
• Advancing Audio Emotion and Intent Recognition with Large Pre-Trained Models and Bayesian Inference
• Overview of Automatic Speech Analysis and Technologies for Neurodegenerative Disorders: Diagnosis and Assistive Applications
• Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices
• Active Learning of Non-semantic Speech Tasks with Pretrained Models