Deep Reinforcement Learning-based Image Captioning with Embedding Reward

Abstract

Image captioning is a challenging problem owing to the complexity in understanding the image content and diverse ways of describing it in natural language. Recent advances in deep neural networks have substantially improved the performance of this task. Most state-of-the-art approaches follow an encoder-decoder framework, which generates captions using a sequential recurrent prediction model. However, in this paper, we introduce a novel decision-making framework for image captioning. We utilize a "policy network" and a "value network" to collaboratively generate captions. The policy network serves as a local guidance by providing the confidence of predicting the next word according to the current state. Additionally, the value network serves as a global and lookahead guidance by evaluating all possible extensions of the current state. In essence, it adjusts the goal of predicting the correct words towards the goal of generating captions similar to the ground truth captions. We train both networks using an actor-critic reinforcement learning model, with a novel reward defined by visual-semantic embedding. Extensive experiments and analyses on the Microsoft COCO dataset show that the proposed framework outperforms state-of-the-art approaches across different evaluation metrics.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Going Deeper with Convolutions
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Sequence Level Training with Recurrent Neural Networks
• Image Captioning with Semantic Attention
• Show and Tell: A Neural Image Caption Generator
• Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• CIDEr: Consensus-based Image Description Evaluation
• Simple Image Description Generator via a Linear Phrase-Based Approach
• Multi-Instance Visual-Semantic Embedding
• Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data

Cited by in corpus (27)

• Deep Reinforcement Learning: An Overview
• AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Discriminability objective for training descriptive captions
• Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning
• Reconstruction Network for Video Captioning
• Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning
• No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
• Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation
• Hierarchical LSTMs with Adaptive Attention for Visual Captioning
• A Comprehensive Survey of Deep Learning for Image Captioning
• Discourse-Aware Neural Rewards for Coherent Text Generation
• Streamlined Dense Video Captioning
• End-to-End Video Captioning with Multitask Reinforcement Learning
• Context and Attribute Grounded Dense Captioning
• Towards Amortized Ranking-Critical Training for Collaborative Filtering
• A Deep Decoder Structure Based on WordEmbedding Regression for An Encoder-Decoder Based Model for Image Captioning
• Improving Reinforcement Learning Based Image Captioning with Natural Language Prior
• Hierarchical Photo-Scene Encoder for Album Storytelling
• Transfer Reward Learning for Policy Gradient-Based Text Generation
• Tell-the-difference: Fine-grained Visual Descriptor via a Discriminating Referee
• Improving Automatic Source Code Summarization via Deep Reinforcement Learning
• FFNet: Video Fast-Forwarding via Reinforcement Learning
• Image Captioning using Multiple Transformers for Self-Attention Mechanism
• Improving Adversarial Text Generation by Modeling the Distant Future
• Deep Reinforcement Learning with Label Embedding Reward for Supervised Image Hashing
• A2-RL: Aesthetics Aware Reinforcement Learning for Image Cropping