Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

Abstract

Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.

ACM Multimedia 2021

References in corpus (12)

• VoxCeleb2: Deep Speaker Recognition
• Squeeze-and-Excitation Networks
• Deep Audio-Visual Speech Recognition
• Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
• In defence of metric learning for speaker recognition
• LRS3-TED: a large-scale dataset for visual speech recognition
• Spot the conversation: speaker diarisation in the wild
• Perfect match: Improved cross-modal embeddings for audio-visual synchronisation
• Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning
• Detecting Aedes Aegypti Mosquitoes through Audio Classification with Convolutional Neural Networks
• End-to-end Audiovisual Speech Activity Detection with Bimodal Recurrent Neural Models
• Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)

Cited by in corpus (11)

• MIntRec: A New Dataset for Multimodal Intent Recognition
• AVA-AVD: Audio-Visual Speaker Diarization in the Wild
• Speaker Extraction with Co-Speech Gestures Cue
• A Deep Learning-Based GPR Forward Solver for Predicting B-Scans of Subsurface Objects
• MM-ALT: A Multimodal Automatic Lyric Transcription System
• Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledge
• EditIQ: Automated Cinematic Editing of Static Wide-Angle Videos via Dialogue Interpretation and Saliency Cues
• Word-level Persian Lipreading Dataset
• FabuLight-ASD: Unveiling Speech Activity via Body Language
• Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions
• SLoClas: A Database for Joint Sound Localization and Classification