Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Abstract

Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduce the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder. The encoder allows one to rank images and sentences while the decoder can generate novel descriptions from scratch. Using LSTM to encode sentences, we match the state-of-the-art performance on Flickr8K and Flickr30K without using object detections. We also set new best results when using the 19-layer Oxford convolutional network. Furthermore we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* - "blue" + "red" is near images of red cars. Sample captions generated for 800 images are made available for comparison.

13 pages. NIPS 2014 deep learning workshop

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Sequence to Sequence Learning with Neural Networks
• Recurrent Neural Network Regularization
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Explain Images with Multimodal Recurrent Neural Networks
• A Multiplicative Model for Learning Distributed Text-Based Attribute Representations

Cited by in corpus (297)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Learning Transferable Visual Models From Natural Language Supervision
• Generative Adversarial Text to Image Synthesis
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• VQA: Visual Question Answering
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• Exploring Models and Data for Image Question Answering
• Generative Moment Matching Networks
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Multi-Modal Knowledge Graph Construction and Application: A Survey
• Grid Long Short-Term Memory
• Doctor AI: Predicting Clinical Events via Recurrent Neural Networks
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Detecting Sarcasm in Multimodal Social Platforms
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• A Comprehensive Survey on Cross-modal Retrieval
• An Actor-Critic Algorithm for Sequence Prediction
• Learning What and Where to Draw
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• Exploring Nearest Neighbor Approaches for Image Captioning
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• Multimodal Machine Learning: A Survey and Taxonomy
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Learning Dual Semantic Relations with Graph Attention for Image-Text Matching
• CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
• Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images
• Generation and Comprehension of Unambiguous Object Descriptions
• Language Models for Image Captioning: The Quirks and What Works
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
• Root Mean Square Layer Normalization
• Coupled Generative Adversarial Networks
• Fluency-Guided Cross-Lingual Image Captioning
• Where to put the Image in an Image Caption Generator
• Finding beans in burgers: Deep semantic-visual embedding with localization
• Order-Embeddings of Images and Language
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
• DenseCap: Fully Convolutional Localization Networks for Dense Captioning
• Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• Does Multimodality Help Human and Machine for Translation and Image Captioning?
• Attention Correctness in Neural Image Captioning
• Generating Images from Captions with Attention
• Learning Language-Visual Embedding for Movie Understanding with Natural-Language
• Visually Grounded Neural Syntax Acquisition
• See, Hear, and Read: Deep Aligned Representations
• Improving performance of recurrent neural network with relu nonlinearity
• Cross-Modal Music Retrieval and Applications: An Overview of Key Methodologies
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• Neural Attention for Image Captioning: Review of Outstanding Methods
• Show, Recall, and Tell: Image Captioning with Recall Mechanism
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Learning Deep Structure-Preserving Image-Text Embeddings
• Modeling Context in Referring Expressions
• Learning to generalize to new compositions in image understanding
• A Dataset for Movie Description
• Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network
• Phrase-based Image Captioning
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Emergent Translation in Multi-Agent Communication
• Video Storytelling: Textual Summaries for Events
• From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge
• Stacked Cross Attention for Image-Text Matching
• Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training
• Dual Attention Networks for Multimodal Reasoning and Matching
• Joint Embedding of Words and Labels for Text Classification
• DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis
• Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text
• Action2Vec: A Crossmodal Embedding Approach to Action Learning
• Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text
• Human Motion Modeling using DVGANs
• Your Negative May not Be True Negative: Boosting Image-Text Matching with False Negative Elimination
• Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
• Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
• Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
• Automatic Spatially-aware Fashion Concept Discovery
• Support-set bottlenecks for video-text representation learning
• 3G structure for image caption generation
• DeepStory: Video Story QA by Deep Embedded Memory Networks
• What value do explicit high level concepts have in vision to language problems?
• Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrieval
• Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Teaching Machines to Describe Images via Natural Language Feedback
• Multi-modal gated recurrent units for image description
• TediGAN: Text-Guided Diverse Face Image Generation and Manipulation
• CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
• Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Learning to Answer Questions From Image Using Convolutional Neural Network
• IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval
• Jointly Modeling Embedding and Translation to Bridge Video and Language
• Simple Image Description Generator via a Linear Phrase-Based Approach
• Image Captioning with Deep Bidirectional LSTMs
• M-VAD Names: a Dataset for Video Captioning with Naming
• Incorporating Global Visual Features into Attention-Based Neural Machine Translation
• Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
• Bilinear Representation for Language-based Image Editing Using Conditional Generative Adversarial Networks
• Natural Language Object Retrieval
• Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning
• Learning Semantic Concepts and Order for Image and Sentence Matching
• Learning Visually-Grounded Semantics from Contrastive Adversarial Samples
• Learning the Best Pooling Strategy for Visual Semantic Embedding
• Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
• Visual Semantic Reasoning for Image-Text Matching
• Learning Two-Branch Neural Networks for Image-Text Matching Tasks
• Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
• A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus
• VISALOGY: Answering Visual Analogy Questions
• SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text
• Semantic Compositional Networks for Visual Captioning
• Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching
• BreakingNews: Article Annotation by Image and Text Processing
• A unified representation network for segmentation with missing modalities
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
• Position Focused Attention Network for Image-Text Matching
• Learning Audio - Sheet Music Correspondences for Score Identification and Offline Alignment
• Cross-modal Hallucination for Few-shot Fine-grained Recognition
• Multimodal Contrastive Training for Visual Representation Learning
• Neural Self Talk: Image Understanding via Continuous Questioning and Answering
• Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
• Multimodal Grounding for Language Processing
• Transitive Hashing Network for Heterogeneous Multimedia Retrieval
• Deep Kernel Learning
• Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
• Saliency-Guided Attention Network for Image-Sentence Matching
• Multilingual Multi-modal Embeddings for Natural Language Processing
• Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognition
• Generating Multi-Sentence Lingual Descriptions of Indoor Scenes
• AMC: Attention guided Multi-modal Correlation Learning for Image Search
• Automatic Concept Discovery from Parallel Text and Visual Corpora
• Large-Scale Visual Relationship Understanding
• Multimodal Hierarchical Reinforcement Learning Policy for Task-Oriented Visual Dialog
• Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
• Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
• Learning to Understand Phrases by Embedding the Dictionary
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions
• Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts
• A Comprehensive Survey of Deep Learning for Image Captioning
• Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
• Content-Based Video-Music Retrieval Using Soft Intra-Modal Structure Constraint
• Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
• More Grounded Image Captioning by Distilling Image-Text Matching Model
• T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval
• A New Evaluation Protocol and Benchmarking Results for Extendable Cross-media Retrieval
• Open-World Visual Recognition Using Knowledge Graphs
• Image-embodied Knowledge Representation Learning
• Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval
• Interactive Image Manipulation with Natural Language Instruction Commands
• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
• Multilingual Visual Sentiment Concept Matching
• Step-Wise Hierarchical Alignment Network for Image-Text Matching
• Deep Embedding for Spatial Role Labeling
• General Partial Label Learning via Dual Bipartite Graph Autoencoder
• Deep Bayesian Active Learning for Multiple Correct Outputs
• AutoCaption: Image Captioning with Neural Architecture Search
• Learning language through pictures
• Stacked Cross-modal Feature Consolidation Attention Networks for Image Captioning
• Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos
• Connectionist-Symbolic Machine Intelligence using Cellular Automata based Reservoir-Hyperdimensional Computing
• Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task
• Instance-aware Image and Sentence Matching with Selective Multimodal LSTM
• HANet: Hierarchical Alignment Networks for Video-Text Retrieval
• Better Text Understanding Through Image-To-Text Transfer
• Video-Text Pre-training with Learned Regions
• Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing
• Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
• Pre-gen metrics: Predicting caption quality metrics without generating captions
• Quantifying the amount of visual information used by neural caption generators
• IDEL: In-Database Entity Linking with Neural Embeddings
• Improving Visually Grounded Sentence Representations with Self-Attention
• Adversarial Attribute-Image Person Re-identification
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
• LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
• Towards General Purpose Vision Systems
• Tensor Product Generation Networks for Deep NLP Modeling
• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
• Text-guided Attention Model for Image Captioning
• Unveiling the Dreams of Word Embeddings: Towards Language-Driven Image Generation
• Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
• Full-Network Embedding in a Multimodal Embedding Pipeline
• Informative Image Captioning with External Sources of Information
• Weakly supervised cross-domain alignment with optimal transport
• Don't only Feel Read: Using Scene text to understand advertisements
• Improving Sales Forecasting Accuracy: A Tensor Factorization Approach with Demand Awareness
• Gaussian Smoothen Semantic Features (GSSF) -- Exploring the Linguistic Aspects of Visual Captioning in Indian Languages (Bengali) Using MSCOCO Framework
• Exploring Uncertainty Measures for Image-Caption Embedding-and-Retrieval Task
• Phrase-based Image Captioning with Hierarchical LSTM Model
• Self-Guiding Multimodal LSTM - when we do not have a perfect training dataset for image captioning
• Image Captioning using Deep Stacked LSTMs, Contextual Word Embeddings and Data Augmentation
• UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations
• Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences
• Learning Soft-Attention Models for Tempo-invariant Audio-Sheet Music Retrieval
• Joint Visual-Textual Embedding for Multimodal Style Search
• Image Captioning with Clause-Focused Metrics in a Multi-Modal Setting for Marketing
• Ladder Loss for Coherent Visual-Semantic Embedding
• HUSE: Hierarchical Universal Semantic Embeddings
• Rethinking movie genre classification with fine-grained semantic clustering
• MHSAN: Multi-Head Self-Attention Network for Visual Semantic Embedding
• Learning to Represent Image and Text with Denotation Graph
• Adaptive Offline Quintuplet Loss for Image-Text Matching
• Learning from Large-scale Noisy Web Data with Ubiquitous Reweighting for Image Classification
• Evaluating Multimodal Representations on Sentence Similarity: vSTS, Visual Semantic Textual Similarity Dataset
• Order embeddings and character-level convolutions for multimodal alignment
• A Visual Embedding for the Unsupervised Extraction of Abstract Semantics
• Learning Temporal Embeddings for Complex Video Analysis
• Lessons learned in multilingual grounded language learning
• Unsupervised Stylish Image Description Generation via Domain Layer Norm
• Generative Imagination Elevates Machine Translation
• Evaluating Text-to-Image Matching using Binary Image Selection (BISON)
• Interpretable Visual Reasoning via Induced Symbolic Space
• Personalized Multimodal Feedback Generation in Education
• A sequential guiding network with attention for image captioning
• 3D Human motion anticipation and classification
• Visual-Texual Emotion Analysis with Deep Coupled Video and Danmu Neural Networks
• HAL: Improved Text-Image Matching by Mitigating Visual Semantic Hubs
• Cross-Modal Generative Augmentation for Visual Question Answering
• Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval
• Face-Cap: Image Captioning using Facial Expression Analysis
• A Better Loss for Visual-Textual Grounding
• Caption Generation on Scenes with Seen and Unseen Object Categories
• Grammar-Based Grounded Lexicon Learning
• FiCo-ITR: bridging fine-grained and coarse-grained image-text retrieval for comparative performance analysis
• Image Captioning with Unseen Objects
• MULE: Multimodal Universal Language Embedding
• Reasoning about Fine-grained Attribute Phrases using Reference Games
• TIGEr: Text-to-Image Grounding for Image Caption Evaluation
• Learning Relation Alignment for Calibrated Cross-modal Retrieval
• More Than Just Attention: Improving Cross-Modal Attentions with Contrastive Constraints for Image-Text Matching
• Exploiting Visual Semantic Reasoning for Video-Text Retrieval
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• Zoom-RNN: A Novel Method for Person Recognition Using Recurrent Neural Networks
• Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
• Big Data driven Product Design: A Survey
• Diverse and Styled Image Captioning Using SVD-Based Mixture of Recurrent Experts
• A Strong and Robust Baseline for Text-Image Matching
• Bilingual is At Least Monolingual (BALM): A Novel Translation Algorithm that Encodes Monolingual Priors
• Target-Oriented Deformation of Visual-Semantic Embedding Space
• Beyond the Deep Metric Learning: Enhance the Cross-Modal Matching with Adversarial Discriminative Domain Regularization
• SSAN: Separable Self-Attention Network for Video Representation Learning
• On Architectures for Including Visual Information in Neural Language Models for Image Description
• Attention as a Perspective for Learning Tempo-invariant Audio Queries
• ActBERT: Learning Global-Local Video-Text Representations
• Leveraging Visual Question Answering for Image-Caption Ranking
• Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matching
• A Universal Model for Cross Modality Mapping by Relational Reasoning
• Rudder: A Cross Lingual Video and Text Retrieval Dataset
• An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information
• Story-oriented Image Selection and Placement
• Upgrading the Newsroom: An Automated Image Selection System for News Articles
• What is Learned in Visually Grounded Neural Syntax Acquisition
• Attribute Guided Sparse Tensor-Based Model for Person Re-Identification
• CanvasGAN: A simple baseline for text to image generation by incrementally patching a canvas
• PUNCH: Positive UNlabelled Classification based information retrieval in Hyperspectral images
• Multimodal Story Generation on Plural Images
• Adversarial Ranking Attack and Defense
• MUTATT: Visual-Textual Mutual Guidance for Referring Expression Comprehension
• Exploring the Challenges towards Lifelong Fact Learning
• MULTIMODAL ANALYSIS: Informed content estimation and audio source separation
• Survey of Visual-Semantic Embedding Methods for Zero-Shot Image Retrieval
• SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval
• Face and Voice Cross-modal Association with Learning Convex Feature Embedding
• Training Efficiency and Robustness in Deep Learning
• Automated Knee X-ray Report Generation
• Image2song: Song Retrieval via Bridging Image Content and Lyric Words
• Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructions
• Learning Articulated Motion Models from Visual and Lingual Signals
• PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling
• Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms
• Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP
• The State of the Art when using GPUs in Devising Image Generation Methods Using Deep Learning
• An animated picture says at least a thousand words: Selecting Gif-based Replies in Multimodal Dialog
• Improving Zero-shot Multilingual Neural Machine Translation for Low-Resource Languages
• Contrastive Learning of Visual-Semantic Embeddings
• Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision
• OptiBox: Breaking the Limits of Proposals for Visual Grounding
• YouMakeup VQA Challenge: Towards Fine-grained Action Understanding in Domain-Specific Videos
• The Long-Short Story of Movie Description
• Retrieving and Highlighting Action with Spatiotemporal Reference
• Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks