Multitask Detection of Speaker Changes, Overlapping Speech and Voice Activity Using wav2vec 2.0

Abstract

Self-supervised learning approaches have lately achieved great success on a broad spectrum of machine learning problems. In the field of speech processing, one of the most successful recent self-supervised models is wav2vec 2.0. In this paper, we explore the effectiveness of this model on three basic speech classification tasks: speaker change detection, overlapped speech detection, and voice activity detection. First, we concentrate on only one task -- speaker change detection -- where our proposed system surpasses the previously reported results on four different corpora, and achieves comparable performance even when trained on out-of-domain data from an artificially designed dataset. Then we expand our approach to tackle all three tasks in a single multitask system with state-of-the-art performance on the AMI corpus. The implementation of the algorithms in this paper is publicly available at https://github.com/mkunes/w2v2_audioFrameClassification.

To appear at ICASSP 2023

References in corpus (4)

• Overlap-aware diarization: resegmentation using neural end-to-end overlapped speech detection
• BW-EDA-EEND: Streaming End-to-End Neural Speaker Diarization for a Variable Number of Speakers
• Detection of Prosodic Boundaries in Speech Using Wav2Vec 2.0
• Sequence-level Speaker Change Detection with Difference-based Continuous Integrate-and-fire