Combination of Deep Speaker Embeddings for Diarisation

Abstract

Significant progress has recently been made in speaker diarisation after the introduction of d-vectors as speaker embeddings extracted from neural network (NN) speaker classifiers for clustering speech segments. To extract better-performing and more robust speaker embeddings, this paper proposes a c-vector method by combining multiple sets of complementary d-vectors derived from systems with different NN components. Three structures are used to implement the c-vectors, namely 2D self-attentive, gated additive, and bilinear pooling structures, relying on attention mechanisms, a gating mechanism, and a low-rank bilinear pooling mechanism respectively. Furthermore, a neural-based single-pass speaker diarisation pipeline is also proposed in this paper, which uses NNs to achieve voice activity detection, speaker change point detection, and speaker embedding extraction. Experiments and detailed analyses are conducted on the challenging AMI and NIST RT05 datasets which consist of real meetings with 4--10 speakers and a wide range of acoustic conditions. For systems trained on the AMI training set, relative speaker error rate (SER) reductions of 13% and 29% are obtained by using c-vectors instead of d-vectors on the AMI dev and eval sets respectively, and a relative reduction of 15% in SER is observed on RT05, which shows the robustness of the proposed methods. By incorporating VoxCeleb data into the training set, the best c-vector system achieved 7%, 17% and16% relative SER reduction compared to the d-vector on the AMI dev, eval, and RT05 sets respectively

arXiv commentManualscript accepted by Neural Networks

References in corpus (12)

• VoxCeleb2: Deep Speaker Recognition
• VoxCeleb: a large-scale speaker identification dataset
• A Structured Self-attentive Sentence Embedding
• Attentive Statistics Pooling for Deep Speaker Embedding
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Hadamard Product for Low-rank Bilinear Pooling
• LSTM based Similarity Measurement with Spectral Clustering for Speaker Diarization
• Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection
• Attention-Based Models for Text-Dependent Speaker Verification
• Speaker Diarization using Deep Recurrent Convolutional Neural Networks for Speaker Embeddings
• Improved Large-margin Softmax Loss for Speaker Diarisation
• Incremental Transfer Learning in Two-pass Information Bottleneck based Speaker Diarization System for Meetings

Cited by in corpus (4)

• Distribution-based Emotion Recognition in Conversation
• Integrating Emotion Recognition with Speech Recognition and Speaker Diarisation for Conversations
• Sequence-level Speaker Change Detection with Difference-based Continuous Integrate-and-fire
• Content-Aware Speaker Embeddings for Speaker Diarisation