Beyond Mimicry: Constrained Exploration with GRPO for Joint Multi-Talker ASR and Diarization under Unknown Speaker Counts
arXiv:2611.22627 · doi:10.21437/Interspeech.2026-2297
Abstract
Joint multi-talker ASR with speaker attribution and timestamps is particularly challenging when the speaker count is unknown and the output must follow a strict structure. We observe that Supervised Fine-Tuning (SFT)-only alignment of SpeechLLMs is brittle under dense overlaps, often leading to burst hallucinations and malformed speaker/timestamp tags. We propose a unified two-stage generative framework: (i) CoT-augmented SFT that infers the global speaker count before transcription, and (ii) GRPO-based constrained exploration guided by a Multi-dimensional Constraint-Aware Reward (MCAR) that directly optimizes permutation-invariant accuracy while enforcing counting, temporal, and structural constraints. On Libri3Mix and a Dynamic-Mix (2+3) set (speaker count unknown at test time), our model achieves 14.52%/9.24% cpWER and 1.95%/1.12% WDER, respectively, with GRPO yielding a 35%/54% relative cpWER reduction over SFT+CoT.
arXiv comment5 pages, Interspeech 2026
References in corpus (9)
- LoRA: Low-Rank Adaptation of Large Language Models
- LibriMix: An Open-Source Dataset for Generalizable Speech Separation
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
- SALMONN: Towards Generic Hearing Abilities for Large Language Models
- Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio
- Qwen3-Omni Technical Report
- Kimi-Audio Technical Report
- Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder