Look, Listen and Learn - A Multimodal LSTM for Speaker Identification

Abstract

Speaker identification refers to the task of localizing the face of a person who has the same identity as the ongoing voice in a video. This task not only requires collective perception over both visual and auditory signals, the robustness to handle severe quality degradations and unconstrained content variations are also indispensable. In this paper, we describe a novel multimodal Long Short-Term Memory (LSTM) architecture which seamlessly unifies both visual and auditory modalities from the beginning of each sequence input. The key idea is to extend the conventional LSTM by not only sharing weights across time steps, but also sharing weights across modalities. We show that modeling the temporal dependency across face and voice can significantly improve the robustness to content quality degradations and variations. We also found that our multimodal LSTM is robustness to distractors, namely the non-speaking identities. We applied our multimodal LSTM to The Big Bang Theory dataset and showed that our system outperforms the state-of-the-art systems in speaker identification with lower false alarm rate and higher recognition accuracy.

arXiv commentThe 30th AAAI Conference on Artificial Intelligence (AAAI-16)

References in corpus (12)

• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Learning Face Representation by Joint Identification-Verification
• Attention-Based Models for Speech Recognition
• DRAW: A Recurrent Neural Network For Image Generation
• DeepID3: Face Recognition with Very Deep Neural Networks
• Skip-Thought Vectors
• Learning Deep Representation for Face Alignment with Auxiliary Attributes
• Neural Responding Machine for Short-Text Conversation
• From Facial Parts Responses to Face Detection: A Deep Learning Approach
• Generative Image Modeling Using Spatial LSTMs
• On Vectorization of Deep Convolutional Neural Networks for Vision Tasks

Cited by in corpus (12)

• Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition
• A Hybrid Method for Traffic Flow Forecasting Using Multimodal Deep Learning
• MGANet: A Robust Model for Quality Enhancement of Compressed Video
• Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning
• Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially-Aware Language Acquisition
• Two-phase Hair Image Synthesis by Self-Enhancing Generative Model
• A Comparison of Pooling Methods on LSTM Models for Rare Acoustic Event Classification
• Self-supervised learning for audio-visual speaker diarization
• Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning
• Sensor Fusion: Gated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data
• Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representations
• Online Multi-modal Person Search in Videos