MAAS: Multi-modal Assignation for Active Speaker Detection
arXiv:2101.03682
Abstract
Active speaker detection requires a solid integration of multi-modal cues. While individual modalities can approximate a solution, accurate predictions can only be achieved by explicitly fusing the audio and visual features and modeling their temporal progression. Despite its inherent muti-modal nature, current methods still focus on modeling and fusing short-term audiovisual features for individual speakers, often at frame level. In this paper we present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem, and provides a straightforward strategy where independent visual features from potential speakers in the scene are assigned to a previously detected speech event. Our experiments show that, an small graph data structure built from a single frame, allows to approximate an instantaneous audio-visual assignment problem. Moreover, the temporal extension of this initial graph achieves a new state-of-the-art on the AVA-ActiveSpeaker dataset with a mAP of 88.8\%.
References in corpus (11)
- Semi-Supervised Classification with Graph Convolutional Networks
- VoxCeleb2: Deep Speaker Recognition
- VoxCeleb: a large-scale speaker identification dataset
- Fast Graph Representation Learning with PyTorch Geometric
- DeeperGCN: All You Need to Train Deeper GCNs
- Spot the conversation: speaker diarisation in the wild
- Neural Predictive Coding using Convolutional Neural Networks towards Unsupervised Learning of Speaker Characteristics
- VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking
- Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)
- SGAS: Sequential Greedy Architecture Search
- Point Clouds Learning with Attention-based Graph Convolution Networks