DenseCap: Fully Convolutional Localization Networks for Dense Captioning

Abstract

We introduce the dense captioning task, which requires a computer vision system to both localize and describe salient regions in images in natural language. The dense captioning task generalizes object detection when the descriptions consist of a single word, and Image Captioning when one predicted region covers the full image. To address the localization and description task jointly we propose a Fully Convolutional Localization Network (FCLN) architecture that processes an image with a single, efficient forward pass, requires no external regions proposals, and can be trained end-to-end with a single round of optimization. The architecture is composed of a Convolutional Network, a novel dense localization layer, and Recurrent Neural Network language model that generates the label sequences. We evaluate our network on the Visual Genome dataset, which comprises 94,000 images and 4,100,000 region-grounded captions. We observe both speed and accuracy improvements over baselines based on current state of the art approaches in both generation and retrieval settings.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Going Deeper with Convolutions
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• DRAW: A Recurrent Neural Network For Image Generation
• Visualizing and Understanding Recurrent Networks
• Explain Images with Multimodal Recurrent Neural Networks
• Learning a Recurrent Visual Representation for Image Caption Generation
• Spatial Transformer Networks
• CIDEr: Consensus-based Image Description Evaluation
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Cited by in corpus (7)

• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach
• iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering
• Dense Image Representation with Spatial Pyramid VLAD Coding of CNN for Locally Robust Captioning
• Wavelet based edge feature enhancement for convolutional neural networks
• Image Captioning and Classification of Dangerous Situations
• Visual Question Answering Using Semantic Information from Image Descriptions