From Captions to Visual Concepts and Back

Abstract

This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our held-out test set, the system captions have equal or better quality 34% of the time.

version corresponding to CVPR15 paper

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• A Convolutional Neural Network for Modelling Sentences
• Explain Images with Multimodal Recurrent Neural Networks
• Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network
• CIDEr: Consensus-based Image Description Evaluation

Cited by in corpus (55)

• Microsoft COCO Captions: Data Collection and Evaluation Server
• VQA: Visual Question Answering
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Fully Connected Deep Structured Networks
• Image Captioning with Semantic Attention
• Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
• Exploring Nearest Neighbor Approaches for Image Captioning
• AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Language Models for Image Captioning: The Quirks and What Works
• Visual Relationship Detection with Language Priors
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Recurrent Neural Networks Hardware Implementation on FPGA
• DenseCap: Fully Convolutional Localization Networks for Dense Captioning
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• Contextual Action Recognition with R*CNN
• CNN+CNN: Convolutional Decoders for Image Captioning
• See, Hear, and Read: Deep Aligned Representations
• Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion
• Detecting Visual Relationships with Deep Relational Networks
• Person Search with Natural Language Description
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• A Dataset for Movie Description
• Natural Language Understanding with Distributed Representation
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators
• Scene Graph Generation from Objects, Phrases and Region Captions
• Simple Image Description Generator via a Linear Phrase-Based Approach
• Weakly Supervised Dense Video Captioning
• Context-aware Captions from Context-agnostic Supervision
• Content-based Video Indexing and Retrieval Using Corr-LDA
• Weakly-supervised learning of visual relations
• Semantic Compositional Networks for Visual Captioning
• Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
• Generating Multi-Sentence Lingual Descriptions of Indoor Scenes
• VQABQ: Visual Question Answering by Basic Questions
• Relaxed Multiple-Instance SVM with Application to Object Discovery
• Learning Visual Knowledge Memory Networks for Visual Question Answering
• A Semi-supervised Framework for Image Captioning
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• Learning language through pictures
• Transform and Tell: Entity-Aware News Image Captioning
• Dense Captioning with Joint Inference and Visual Context
• Dual-Glance Model for Deciphering Social Relationships
• Commonly Uncommon: Semantic Sparsity in Situation Recognition
• Learning to Disambiguate by Asking Discriminative Questions
• ViP-CNN: Visual Phrase Guided Convolutional Neural Network
• Natural Vocabulary Emerges from Free-Form Annotations
• Segment-Phrase Table for Semantic Segmentation, Visual Entailment and Paraphrasing
• cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey
• Automated Image Captioning for Rapid Prototyping and Resource Constrained Environments