SUPERB: Speech processing Universal PERformance Benchmark

Abstract

Self-supervised learning (SSL) has proven vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the speech processing community lacks a similar setup to systematically explore the paradigm. To bridge this gap, we introduce Speech processing Universal PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data. Among multiple usages of the shared model, we especially focus on extracting the representation learned from SSL due to its preferable re-usability. We present a simple framework to solve SUPERB tasks by learning task-specialized lightweight prediction heads on top of the frozen shared model. Our results demonstrate that the framework is promising as SSL representations show competitive generalizability and accessibility across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a benchmark toolkit to fuel the research in representation learning and general speech processing.

arXiv commentTo appear in Interspeech 2021

References in corpus (10)

• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
• LibriMix: An Open-Source Dataset for Generalizable Speech Separation
• wav2letter++: The Fastest Open-source Speech Recognition System
• Towards Learning a Universal Non-Semantic Representation of Speech
• DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
• Exploring wav2vec 2.0 on speaker verification and language identification
• How Useful is Self-Supervised Pretraining for Visual Tasks?

Cited by in corpus (13)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• SpeechBrain: A General-Purpose Speech Toolkit
• BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations
• Disentangled Speech Representation Learning Based on Factorized Hierarchical Variational Autoencoder with Self-Supervised Objective
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models
• Transfer Learning for Multi-lingual Tasks -- a Survey
• Deep Spectro-temporal Artifacts for Detecting Synthesized Speech
• Towards Learning Universal Audio Representations
• SVEva Fair: A Framework for Evaluating Fairness in Speaker Verification
• Do We Still Need Automatic Speech Recognition for Spoken Language Understanding?
• Pretext Tasks selection for multitask self-supervised speech representation learning
• Speaker Group Encoding in Self-supervised Speech Recognition Models