See, Hear, and Read: Deep Aligned Representations

Abstract

We capitalize on large amounts of readily-available, synchronous data to learn a deep discriminative representations shared across three major natural modalities: vision, sound and language. By leveraging over a year of sound from video and millions of sentences paired with images, we jointly train a deep convolutional network for aligned representation learning. Our experiments suggest that this representation is useful for several tasks, such as cross-modal retrieval or transferring classifiers between modalities. Moreover, although our network is only trained with image+text and image+sound pairs, it can transfer between text and sound as well, a transfer the network never observed during training. Visualizations of our representation reveal many hidden units which automatically emerge to detect concepts, independent of the modality.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Object Detectors Emerge in Deep Scene CNNs
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• SoundNet: Learning Sound Representations from Unlabeled Video
• Show and Tell: A Neural Image Caption Generator
• From Captions to Visual Concepts and Back
• Cross Modal Distillation for Supervision Transfer

Cited by in corpus (22)

• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Self-Supervised MultiModal Versatile Networks
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• Self-supervised Audiovisual Representation Learning for Remote Sensing Data
• Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision
• Self-supervised Moving Vehicle Tracking with Stereo Sound
• Deep Audio-Visual Learning: A Survey
• Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
• Weakly Supervised Representation Learning for Unsynchronized Audio-Visual Events
• Uncertainty aware audiovisual activity recognition using deep Bayesian variational inference
• SoundSpaces: Audio-Visual Navigation in 3D Environments
• Deep Multimodal Feature Encoding for Video Ordering
• Grounding Object Detections With Transcriptions
• Self-supervised Audio Spatialization with Correspondence Classifier
• New Ideas and Trends in Deep Multimodal Content Understanding: A Review
• Multilevel Language and Vision Integration for Text-to-Clip Retrieval
• Assessing the Contribution of Semantic Congruency to Multisensory Integration and Conflict Resolution
• XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification
• Unsupervised Generative Adversarial Alignment Representation for Sheet music, Audio and Lyrics
• Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications
• Do We Need Sound for Sound Source Localization?
• Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and Videos