On Learning Associations of Faces and Voices

Abstract

In this paper, we study the associations between human faces and voices. Audiovisual integration, specifically the integration of facial and vocal information is a well-researched area in neuroscience. It is shown that the overlapping information between the two modalities plays a significant role in perceptual tasks such as speaker identification. Through an online study on a new dataset we created, we confirm previous findings that people can associate unseen faces with corresponding voices and vice versa with greater than chance accuracy. We computationally model the overlapping information between faces and voices and show that the learned cross-modal representation contains enough information to identify matching faces and voices with performance similar to that of humans. Our representation exhibits correlations to certain demographic attributes and features obtained from either visual or aural modality alone. We release our dataset of audiovisual recordings and demographic annotations of people reading out short text used in our studies.

27 pages including the supplementary material; Accepted to ACCV 2018

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Adversarial Discriminative Domain Adaptation
• SoundNet: Learning Sound Representations from Unlabeled Video
• Order-Embeddings of Images and Language
• Putting a Face to the Voice: Fusing Audio and Visual Signals Across a Video to Determine Speakers

Cited by in corpus (12)

• Perfect match: Improved cross-modal embeddings for audio-visual synchronisation
• Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision
• Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
• VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
• Voice-Face Cross-modal Matching and Retrieval: A Benchmark
• FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection
• Speech Fusion to Face: Bridging the Gap Between Human's Vocal Characteristics and Facial Imaging
• Multimodal Target Speech Separation with Voice and Face References
• Reconstructing faces from voices
• Controlled AutoEncoders to Generate Faces from Voices
• Noise-tolerant Audio-visual Online Person Verification using an Attention-based Neural Network Fusion
• Active Speakers in Context