ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification

Abstract

Current speaker verification techniques rely on a neural network to extract speaker representations. The successful x-vector architecture is a Time Delay Neural Network (TDNN) that applies statistics pooling to project variable-length utterances into fixed-length speaker characterizing embeddings. In this paper, we propose multiple enhancements to this architecture based on recent trends in the related fields of face verification and computer vision. Firstly, the initial frame layers can be restructured into 1-dimensional Res2Net modules with impactful skip connections. Similarly to SE-ResNet, we introduce Squeeze-and-Excitation blocks in these modules to explicitly model channel interdependencies. The SE block expands the temporal context of the frame layer by rescaling the channels according to global properties of the recording. Secondly, neural networks are known to learn hierarchical features, with each layer operating on a different level of complexity. To leverage this complementary information, we aggregate and propagate features of different hierarchical levels. Finally, we improve the statistics pooling module with channel-dependent frame attention. This enables the network to focus on different subsets of frames during each of the channel's statistics estimation. The proposed ECAPA-TDNN architecture significantly outperforms state-of-the-art TDNN based systems on the VoxCeleb test sets and the 2019 VoxCeleb Speaker Recognition Challenge.

proceedings of INTERSPEECH 2020

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Res2Net: A New Multi-scale Backbone Architecture
• VoxCeleb2: Deep Speaker Recognition
• Squeeze-and-Excitation Networks
• VoxCeleb: a large-scale speaker identification dataset
• Attentive Statistics Pooling for Deep Speaker Embedding
• BUT System Description to VoxCeleb Speaker Recognition Challenge 2019
• VoxSRC 2019: The first VoxCeleb Speaker Recognition Challenge
• Margin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognition
• Cross-Lingual Speaker Verification with Domain-Balanced Hard Prototype Mining and Language-Dependent Score Normalization
• Short-duration Speaker Verification (SdSV) Challenge 2021: the Challenge Evaluation Plan
• Multi-Level and Multi-Scale Feature Aggregation Using Sample-level Deep Convolutional Neural Networks for Music Classification

Cited by in corpus (115)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• SpeechBrain: A General-Purpose Speech Toolkit
• Powerset multi-class cross entropy loss for neural speaker diarization
• Fine-tuning wav2vec2 for speaker recognition
• ECAPA-TDNN Embeddings for Speaker Diarization
• A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding
• VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge
• Integrating Frequency Translational Invariance in TDNNs and Frequency Positional Information in 2D ResNets to Enhance Speaker Verification
• UR Channel-Robust Synthetic Speech Detection System for ASVspoof 2021
• The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verification
• An Overview of Indian Spoken Language Recognition from Machine Learning Perspective
• Semi-Supervised Contrastive Learning with Generalized Contrastive Loss and Its Application to Speaker Recognition
• Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation
• Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus
• Attention Back-end for Automatic Speaker Verification with Multiple Enrollment Utterances
• C3-DINO: Joint Contrastive and Non-contrastive Self-Supervised Learning for Speaker Verification
• The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020
• WavThruVec: Latent speech representation as intermediate features for neural speech synthesis
• The VoxCeleb Speaker Recognition Challenge: A Retrospective
• NeXt-TDNN: Modernizing Multi-Scale Temporal Convolution Backbone for Speaker Verification
• EfficientTDNN: Efficient Architecture Search for Speaker Recognition
• The Influence of Dataset Partitioning on Dysfluency Detection Systems
• Cross-Lingual Speaker Verification with Domain-Balanced Hard Prototype Mining and Language-Dependent Score Normalization
• Revealing Emotional Clusters in Speaker Embeddings: A Contrastive Learning Strategy for Speech Emotion Recognition
• Effects of language mismatch in automatic forensic voice comparison using deep learning embeddings
• Towards Robust Family-Infant Audio Analysis Based on Unsupervised Pretraining of Wav2vec 2.0 on Large-Scale Unlabeled Family Audio
• PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings
• Decoupling and Interacting Multi-Task Learning Network for Joint Speech and Accent Recognition
• Representation Selective Self-distillation and wav2vec 2.0 Feature Exploration for Spoof-aware Speaker Verification
• The IDLAB VoxCeleb Speaker Recognition Challenge 2020 System Description
• USAT: A Universal Speaker-Adaptive Text-to-Speech Approach
• Improving fairness in speaker verification via Group-adapted Fusion Network
• Adversarial speech for voice privacy protection from Personalized Speech generation
• Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations
• Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verification
• Analysis of constant-Q filterbank based representations for speech emotion recognition
• Fake the Real: Backdoor Attack on Deep Speech Classification via Voice Conversion
• Exploiting Pre-Trained ASR Models for Alzheimer's Disease Recognition Through Spontaneous Speech
• Generalizing Speaker Verification for Spoof Awareness in the Embedding Space
• GMM-ResNet2: Ensemble of Group ResNet Networks for Synthetic Speech Detection
• The ByteDance Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2021
• Training speaker recognition systems with limited data
• The xx205 System for the VoxCeleb Speaker Recognition Challenge 2020
• Towards Lightweight Speaker Verification via Adaptive Neural Network Quantization
• Introducing Model Inversion Attacks on Automatic Speaker Recognition
• Cluster-Guided Unsupervised Domain Adaptation for Deep Speaker Embedding
• TSUP Speaker Diarization System for Conversational Short-phrase Speaker Diarization Challenge
• SLMIA-SR: Speaker-Level Membership Inference Attacks against Speaker Recognition Systems
• Cosine Scoring with Uncertainty for Neural Speaker Embedding
• Spectrogram features for audio and speech analysis
• HLT-NUS SUBMISSION FOR 2020 NIST Conversational Telephone Speech SRE
• On the Impact of Voice Anonymization on Speech Diagnostic Applications: a Case Study on COVID-19 Detection
• LMD: A Learnable Mask Network to Detect Adversarial Examples for Speaker Verification
• AISPEECH-SJTU accent identification system for the Accented English Speech Recognition Challenge
• Tackling the Score Shift in Cross-Lingual Speaker Verification by Exploiting Language Information
• Analysis of Speech Temporal Dynamics in the Context of Speaker Verification and Voice Anonymization
• FlexLip: A Controllable Text-to-Lip System
• Learning Audio-Visual Dereverberation
• Improving Speaker Identification for Shared Devices by Adapting Embeddings to Speaker Subsets
• Diffusion-Based Adversarial Purification for Speaker Verification
• Confidence-based Ensembles of End-to-End Speech Recognition Models
• Privacy-oriented manipulation of speaker representations
• VoicePAT: An Efficient Open-source Evaluation Toolkit for Voice Privacy Research
• Data-Efficient Low-Complexity Acoustic Scene Classification via Distilling and Progressive Pruning
• Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development
• Optimizing Multi-Taper Features for Deep Speaker Verification
• Exploring Deep Learning for Joint Audio-Visual Lip Biometrics
• Spoofing-Aware Attention based ASV Back-end with Multiple Enrollment Utterances and a Sampling Strategy for the SASV Challenge 2022
• t-EER: Parameter-Free Tandem Evaluation of Countermeasures and Biometric Comparators
• Residual Information in Deep Speaker Embedding Architectures
• North America Bixby Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2021
• A Teacher-Student approach for extracting informative speaker embeddings from speech mixtures
• Complementing Handcrafted Features with Raw Waveform Using a Light-weight Auxiliary Model
• GMM-ResNext: Combining Generative and Discriminative Models for Speaker Verification
• WavRx: a Disease-Agnostic, Generalizable, and Privacy-Preserving Speech Health Diagnostic Model
• HeightCeleb - an enrichment of VoxCeleb dataset with speaker height information
• Building Bilingual and Code-Switched Voice Conversion with Limited Training Data Using Embedding Consistency Loss
• SpeechNAS: Towards Better Trade-off between Latency and Accuracy for Large-Scale Speaker Verification
• Rep Works in Speaker Verification
• Low-Resource Text-to-Speech Synthesis Using Noise-Augmented Training of ForwardTacotron
• Self-supervised Reflective Learning through Self-distillation and Online Clustering for Speaker Representation Learning
• Text-Aware Adapter for Few-Shot Keyword Spotting
• End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis
• The Impact of Automatic Speech Transcription on Speaker Attribution
• Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
• Quantifying Source Speaker Leakage in One-to-One Voice Conversion
• Y-Vector: Multiscale Waveform Encoder for Speaker Embedding
• Towards Reliable Audio Deepfake Attribution and Model Recognition: A Multi-Level Autoencoder-Based Framework
• Adaptive Margin Circle Loss for Speaker Verification
• Two Methods for Spoofing-Aware Speaker Verification: Multi-Layer Perceptron Score Fusion Model and Integrated Embedding Projector
• Enkidu: Universal Frequential Perturbation for Real-Time Audio Privacy Protection against Voice Deepfakes
• MACCIF-TDNN: Multi aspect aggregation of channel and context interdependence features in TDNN-based speaker verification
• An Investigation of Reprogramming for Cross-Language Adaptation in Speaker Verification Systems
• Poformer: A simple pooling transformer for speaker verification
• Self-Tuning Spectral Clustering for Speaker Diarization
• Cooperative Multi-Agent Reinforcement Learning for Adaptive Aggregation in Semi-Supervised Federated Learning with non-IID Data
• Hyperbolic Additive Margin Softmax with Hierarchical Information for Speaker Verification
• XM-ALIGN: Unified Cross-Modal Embedding Alignment for Face-Voice Association
• An Exploration of ECAPA-TDNN and x-vector Speaker Representations in Zero-shot Multi-speaker TTS
• Self-Supervised Frameworks for Speaker Verification via Bootstrapped Positive Sampling
• Long-Term Conversation Analysis: Privacy-Utility Trade-off under Noise and Reverberation
• Robust Speech Activity Detection in the Presence of Singing Voice
• Weakly Supervised Phonological Features for Pathological Speech Analysis
• SoCov: Semi-Orthogonal Parametric Pooling of Covariance Matrix for Speaker Recognition
• Complete and separate: Conditional separation with missing target source attribute completion
• Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech
• Dynamic Multi-scale Convolution for Dialect Identification
• Xi-Vector Embedding for Speaker Recognition
• XMUSPEECH System for VoxCeleb Speaker Recognition Challenge 2021
• Improving Text-Independent Speaker Verification with Auxiliary Speakers Using Graph
• Duality Temporal-channel-frequency Attention Enhanced Speaker Representation Learning
• STC speaker recognition systems for the NIST SRE 2021
• Coupling a generative model with a discriminative learning framework for speaker verification
• Bidirectional Multiscale Feature Aggregation for Speaker Verification
• Face and Voice Cross-modal Association with Learning Convex Feature Embedding