Boosting Image Captioning with Attributes
arXiv:1611.01646
Abstract
Automatically describing an image with a natural language has been an emerging challenge in both fields of computer vision and natural language processing. In this paper, we present Long Short-Term Memory with Attributes (LSTM-A) - a novel architecture that integrates attributes into the successful Convolutional Neural Networks (CNNs) plus Recurrent Neural Networks (RNNs) image captioning framework, by training them in an end-to-end manner. To incorporate attributes, we construct variants of architectures by feeding image representations and attributes into RNNs in different ways to explore the mutual but also fuzzy relationship between them. Extensive experiments are conducted on COCO image captioning dataset and our framework achieves superior results when compared to state-of-the-art deep models. Most remarkably, we obtain METEOR/CIDEr-D of 25.2%/98.6% on testing data of widely used and publicly available splits in (Karpathy & Fei-Fei, 2015) when extracting image representations by GoogleNet and achieve to date top-1 performance on COCO captioning Leaderboard.
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Explain Images with Multimodal Recurrent Neural Networks
- Image Captioning with Semantic Attention
- Language Models for Image Captioning: The Quirks and What Works
- Rich Image Captioning in the Wild
- Watch What You Just Said: Image Captioning with Text-Conditional Attention
Cited by in corpus (24)
- Improved Image Captioning via Policy Gradient optimization of SPIDEr
- AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding
- Fluency-Guided Cross-Lingual Image Captioning
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- MAttNet: Modular Attention Network for Referring Expression Comprehension
- CNN+CNN: Convolutional Decoders for Image Captioning
- Discriminability objective for training descriptive captions
- Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
- End-to-End Dense Video Captioning with Masked Transformer
- Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning
- A Neural Compositional Paradigm for Image Captioning
- Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
- Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
- Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present
- Real-Time Referring Expression Comprehension by Single-Stage Grounding Network
- Fast Image Caption Generation with Position Alignment
- Knowledge driven Description Synthesis for Floor Plan Interpretation
- From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
- VrR-VG: Refocusing Visually-Relevant Relationships
- OBJ2TEXT: Generating Visually Descriptive Language from Object Layouts
- Neural Aesthetic Image Reviewer
- Contextual Local Explanation for Black Box Classifiers
- Grounded Video Description
- Recurrent Models for Situation Recognition