PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings
arXiv:2403.02288 · doi:10.21437/odyssey.2024-17
Abstract
A major drawback of supervised speech separation (SSep) systems is their reliance on synthetic data, leading to poor real-world generalization. Mixture invariant training (MixIT) was proposed as an unsupervised alternative that uses real recordings, yet struggles with overseparation and adapting to long-form audio. We introduce PixIT, a joint approach that combines permutation invariant training (PIT) for speaker diarization (SD) and MixIT for SSep. With a small extra requirement of needing SD labels, it solves the problem of overseparation and allows stitching local separated sources leveraging existing work on clustering-based neural SD. We measure the quality of the separated sources via applying automatic speech recognition (ASR) systems to them. PixIT boosts the performance of various ASR systems across two meeting corpora both in terms of the speaker-attributed and utterance-based word error rates while not requiring any fine-tuning.
Speaker Odyssey 2024
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
- ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
- Robust Speech Recognition via Large-Scale Weak Supervision
- SpeechBrain: A General-Purpose Speech Toolkit
- NeMo: a toolkit for building AI applications using Neural Modules
- Powerset multi-class cross entropy loss for neural speaker diarization
- Unsupervised Sound Separation Using Mixture Invariant Training
- Asteroid: the PyTorch-based audio source separation toolkit for researchers
- TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings
- On Word Error Rate Definitions and their Efficient Computation for Multi-Speaker Speech Recognition Systems
- M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge