Perfect match: Improved cross-modal embeddings for audio-visual synchronisation

Abstract

This paper proposes a new strategy for learning powerful cross-modal embeddings for audio-to-video synchronization. Here, we set up the problem as one of cross-modal retrieval, where the objective is to find the most relevant audio segment given a short video clip. The method builds on the recent advances in learning representations from cross-modal self-supervision. The main contributions of this paper are as follows: (1) we propose a new learning strategy where the embeddings are learnt via a multi-way matching problem, as opposed to a binary classification (matching or non-matching) problem as proposed by recent papers; (2) we demonstrate that performance of this method far exceeds the existing baselines on the synchronization task; (3) we use the learnt embeddings for visual speech recognition in self-supervision, and show that the performance matches the representations learnt end-to-end in a fully-supervised manner.

arXiv commentPreprint. Work in progress

References in corpus (3)

• Lip Reading Sentences in the Wild
• On Learning Associations of Faces and Voices
• Dynamic Temporal Alignment of Speech to Lips

Cited by in corpus (19)

• What Makes for Good Views for Contrastive Learning?
• Spot the conversation: speaker diarisation in the wild
• Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
• Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning
• LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
• AVA-AVD: Audio-Visual Speaker Diarization in the Wild
• Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)
• Speaker Extraction with Co-Speech Gestures Cue
• Deep Audio-Visual Learning: A Survey
• Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild
• VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
• Automatic audiovisual synchronisation for ultrasound tongue imaging
• Active Speaker Detection as a Multi-Objective Optimization with Uncertainty-based Multimodal Fusion
• Detecting Adversarial Attacks On Audiovisual Speech Recognition
• Who said that?: Audio-visual speaker diarisation of real-world meetings
• End-To-End Audiovisual Feature Fusion for Active Speaker Detection
• Look Who's Talking: Active Speaker Detection in the Wild
• Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors
• AlignNet: A Unifying Approach to Audio-Visual Alignment