Deep Visual-Semantic Alignments for Generating Image Descriptions

Abstract

We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Recurrent Neural Network Regularization
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Going Deeper with Convolutions
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Explain Images with Multimodal Recurrent Neural Networks
• Learning a Recurrent Visual Representation for Image Caption Generation
• A Joint Model of Language and Perception for Grounded Attribute Learning
• From Captions to Visual Concepts and Back
• CIDEr: Consensus-based Image Description Evaluation
• Video In Sentences Out

Cited by in corpus (78)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• VQA: Visual Question Answering
• An efficient framework for learning sentence representations
• Fully Connected Deep Structured Networks
• Image Captioning with Semantic Attention
• Describing Videos by Exploiting Temporal Structure
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• Learning Contextual Dependencies with Convolutional Hierarchical Recurrent Neural Networks
• Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
• Language Models for Image Captioning: The Quirks and What Works
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents
• Discriminability objective for training descriptive captions
• Person Search with Natural Language Description
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• A Dataset for Movie Description
• Natural Language Understanding with Distributed Representation
• Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems
• Visual Causal Feature Learning
• Accurate Single Stage Detector Using Recurrent Rolling Convolution
• Using Deep Learning to Localize Gravitational Wave Sources
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Monocular Object Instance Segmentation and Depth Ordering with CNNs
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Learning Deep Neural Networks for Vehicle Re-ID with Visual-spatio-temporal Path Proposals
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
• Simple Image Description Generator via a Linear Phrase-Based Approach
• End-to-end people detection in crowded scenes
• Weakly Supervised Dense Video Captioning
• Natural Language Object Retrieval
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• Context-aware Captions from Context-agnostic Supervision
• Learning the Best Pooling Strategy for Visual Semantic Embedding
• Natural Language Guided Visual Relationship Detection
• Content-based Video Indexing and Retrieval Using Corr-LDA
• Multi-Cue Zero-Shot Learning with Strong Supervision
• GuessWhat?! Visual object discovery through multi-modal dialogue
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
• Semantic Compositional Networks for Visual Captioning
• Stories for Images-in-Sequence by using Visual and Narrative Components
• Generating Descriptions with Grounded and Co-Referenced People
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
• Tartan: Accelerating Fully-Connected and Convolutional Layers in Deep Learning Networks by Exploiting Numerical Precision Variability
• Generating Multi-Sentence Lingual Descriptions of Indoor Scenes
• FlipDial: A Generative Model for Two-Way Visual Dialogue
• VQABQ: Visual Question Answering by Basic Questions
• Image denoising and restoration with CNN-LSTM Encoder Decoder with Direct Attention
• A Semi-supervised Framework for Image Captioning
• An Effective Automatic Image Annotation Model Via Attention Model and Data Equilibrium
• Differential Angular Imaging for Material Recognition
• Learning language through pictures
• Transform and Tell: Entity-Aware News Image Captioning
• UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
• Deep Structured Scene Parsing by Learning with Image Descriptions
• Top-down Visual Saliency Guided by Captions
• Sketch2code: Generating a website from a paper mockup
• Multimodal Learning for Hateful Memes Detection
• Commonly Uncommon: Semantic Sparsity in Situation Recognition
• Hard to Cheat: A Turing Test based on Answering Questions about Images
• Topos and Stacks of Deep Neural Networks
• DeepDiary: Automatic Caption Generation for Lifelogging Image Streams
• Learning to Disambiguate by Asking Discriminative Questions
• "What happens if..." Learning to Predict the Effect of Forces in Images
• Unsupervised Semantic Action Discovery from Video Collections
• Multi-Task Spatiotemporal Neural Networks for Structured Surface Reconstruction
• Memory Matters: Convolutional Recurrent Neural Network for Scene Text Recognition
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
• Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip
• Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language Queries
• Deep Learning Applied to Image and Text Matching
• Target-Tailored Source-Transformation for Scene Graph Generation
• Visual Question Answering Using Semantic Information from Image Descriptions
• Multimodal Story Generation on Plural Images