Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation
arXiv:2309.03019 · doi:10.1109/TASLP.2024.3419426
Abstract
This paper explores the use of ASR-pretrained Conformers for speaker verification, leveraging their strengths in modeling speech signals. We introduce three strategies: (1) Transfer learning to initialize the speaker embedding network, improving generalization and reducing overfitting. (2) Knowledge distillation to train a more flexible speaker verification model, incorporating frame-level ASR loss as an auxiliary task. (3) A lightweight speaker adaptor for efficient feature conversion without altering the original ASR Conformer, allowing parallel ASR and speaker verification. Experiments on VoxCeleb show significant improvements: transfer learning yields a 0.48% EER, knowledge distillation results in a 0.43% EER, and the speaker adaptor approach, with just an added 4.92M parameters to a 130.94M-parameter model, achieves a 0.57% EER. Overall, our methods effectively transfer ASR capabilities to speaker verification tasks.
Published in: IEEE/ACM Transactions on Audio, Speech, and Language Processing
References in corpus (12)
- Res2Net: A New Multi-scale Backbone Architecture
- VoxCeleb2: Deep Speaker Recognition
- VoxCeleb: a large-scale speaker identification dataset
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
- ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
- Attentive Statistics Pooling for Deep Speaker Embedding
- In defence of metric learning for speaker recognition
- NeMo: a toolkit for building AI applications using Neural Modules
- Fine-tuning wav2vec2 for speaker recognition
- The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verification
- The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020
- Robust Speaker Recognition with Transformers Using wav2vec 2.0