Speaker Group Encoding in Self-supervised Speech Recognition Models

Abstract

We investigate what self-supervised speech recognition models (S3Ms) learn about speaker groups (SGs). We examine several states of S3Ms: pretrained, finetuned on speaker identification (SID), finetuned on automatic speech recognition (ASR), and ASR-finetuned using a fairness enhancing algorithm. We find that S3Ms encode information about several speaker group categories (SGCs), including their gender, age, dialect, ethnicity, and whether they are a native speaker. We find that finetuning for SID amplifies certain SGCs, namely those whose variance is more phonetic in nature, though it does not amplify other SGCs, namely those whose variance is more semantic in nature. On the other hand, finetuning for ASR discards phonetically variant speaker group information (SGI) but retains semantically variant SGI. We find that ASR algorithms designed for fairness improvement change to what extent SGI is encoded in S3Ms; however, this is primarily true for for phonetically variant SGCs, and less true for semantically variant SGCs. We discuss how SGI is encoded by each layer, and identify subdimensions of embeddings responsible for encoding different SGCs. Finally, we discuss how our findings could be beneficial in designing fairer ASR algorithms.

References in corpus (17)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• SpeechBrain: A General-Purpose Speech Toolkit
• SUPERB: Speech processing Universal PERformance Benchmark
• See what I'm saying? Comparing Intelligent Personal Assistant use for Native and Non-Native Language Speakers
• Speaker Verification using Convolutional Neural Networks
• Accent-Robust Automatic Speech Recognition Using Supervised and Unsupervised Wav2vec Embeddings
• Silence is Sweeter Than Speech: Self-Supervised Model Using Silence to Store Speaker Information
• Enhancing and Adversarial: Improve ASR with Speaker Labels
• What Do Self-Supervised Speech Models Know About Words?
• Towards measuring fairness in speech recognition: Fair-Speech dataset
• Similarity Analysis of Self-Supervised Speech Representations
• Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing
• Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models
• Self-supervised Predictive Coding Models Encode Speaker and Phonetic Information in Orthogonal Subspaces
• Open Implementation and Study of BEST-RQ for Speech Processing
• Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations
• Less Forgetting for Better Generalization: Exploring Continual-learning Fine-tuning Methods for Speech Self-supervised Representations