Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision

Abstract

The goal of this work is to train discriminative cross-modal embeddings without access to manually annotated data. Recent advances in self-supervised learning have shown that effective representations can be learnt from natural cross-modal synchrony. We build on earlier work to train embeddings that are more discriminative for uni-modal downstream tasks. To this end, we propose a novel training strategy that not only optimises metrics across modalities, but also enforces intra-class feature separation within each of the modalities. The effectiveness of the method is demonstrated on two downstream tasks: lip reading using the features trained on audio-visual synchronisation, and speaker recognition using the features trained for cross-modal biometric matching. The proposed method outperforms state-of-the-art self-supervised baselines by a signficant margin.

arXiv commentUnder submission as a conference paper

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• VoxCeleb2: Deep Speaker Recognition
• VoxCeleb: a large-scale speaker identification dataset
• Contrastive Multiview Coding
• In defence of metric learning for speaker recognition
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• On Learning Associations of Faces and Voices
• Signs in time: Encoding human motion as a temporal image

Cited by in corpus (8)

• VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
• Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification
• An iterative framework for self-supervised deep speaker representation learning
• Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning
• Neural Dubber: Dubbing for Videos According to Scripts
• Look Who's Talking: Active Speaker Detection in the Wild
• LiRA: Learning Visual Speech Representations from Audio through Self-supervision
• Unsupervised Representation Learning for Speaker Recognition via Contrastive Equilibrium Learning