Spot the conversation: speaker diarisation in the wild

Abstract

The goal of this paper is speaker diarisation of videos collected 'in the wild'. We make three key contributions. First, we propose an automatic audio-visual diarisation method for YouTube videos. Our method consists of active speaker detection using audio-visual methods and speaker verification using self-enrolled speaker models. Second, we integrate our method into a semi-automatic dataset creation pipeline which significantly reduces the number of hours required to annotate videos with diarisation labels. Finally, we use this pipeline to create a large-scale diarisation dataset called VoxConverse, collected from 'in the wild' videos, which we will release publicly to the research community. Our dataset consists of overlapping speech, a large and diverse speaker pool, and challenging background conditions.

The dataset will be available for download from http://www.robots.ox.ac.uk/~vgg/data/voxceleb/voxconverse.html . The development set will be released in July 2020, and the test set will be released in October 2020

References in corpus (1)

• In defence of metric learning for speaker recognition

Cited by in corpus (24)

• Visual Speech Recognition for Multiple Languages in the Wild
• Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
• Powerset multi-class cross entropy loss for neural speaker diarization
• VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge
• UniCon: Unified Context Network for Robust Active Speaker Detection
• AVA-AVD: Audio-Visual Speaker Diarization in the Wild
• The VoxCeleb Speaker Recognition Challenge: A Retrospective
• The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020
• USTC-NELSLIP System Description for DIHARD-III Challenge
• The DKU-DukeECE-Lenovo System for the Diarization Task of the 2021 VoxCeleb Speaker Recognition Challenge
• VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
• The ByteDance Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2021
• MAAS: Multi-modal Assignation for Active Speaker Detection
• FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection
• A Survey on Speech Large Language Models for Understanding
• Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020
• Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development
• North America Bixby Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2021
• Towards Measuring and Scoring Speaker Diarization Fairness
• FabuLight-ASD: Unveiling Speech Activity via Body Language
• Look Who's Talking: Active Speaker Detection in the Wild
• The HUAWEI Speaker Diarisation System for the VoxCeleb Speaker Diarisation Challenge
• XMUSPEECH System for VoxCeleb Speaker Recognition Challenge 2021
• EML System Description for VoxCeleb Speaker Diarization Challenge 2020