Review Networks for Caption Generation

Abstract

We propose a novel extension of the encoder-decoder framework, called a review network. The review network is generic and can enhance any existing encoder- decoder model: in this paper, we consider RNN decoders with both CNN and RNN encoders. The review network performs a number of review steps with attention mechanism on the encoder hidden states, and outputs a thought vector after each review step; the thought vectors are used as the input of the attention mechanism in the decoder. We show that conventional encoder-decoders are a special case of our framework. Empirically, we show that our framework improves over state-of- the-art encoder-decoder systems on the tasks of image captioning and source code captioning.

arXiv commentNIPS 2016

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Adaptive Computation Time for Recurrent Neural Networks
• Image Captioning with Semantic Attention
• Structured Generative Models of Natural Source Code

Cited by in corpus (31)

• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• Discriminability objective for training descriptive captions
• Modality-Agnostic Attention Fusion for visual search with text feedback
• Words or Characters? Fine-grained Gating for Reading Comprehension
• 3G structure for image caption generation
• Semi-Supervised QA with Generative Domain-Adaptive Nets
• Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present
• Partially-Supervised Image Captioning
• Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
• Semantic Compositional Networks for Visual Captioning
• A Semi-supervised Framework for Image Captioning
• MAT: A Multimodal Attentive Translator for Image Captioning
• Vision and Language: from Visual Perception to Content Creation
• "Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention
• Video Captioning with Guidance of Multimodal Latent Topics
• Exploring Overall Contextual Information for Image Captioning in Human-Like Cognitive Style
• Reflective Decoding Network for Image Captioning
• Image Captioning with Context-Aware Auxiliary Guidance
• Informative Image Captioning with External Sources of Information
• Phrase-based Image Captioning with Hierarchical LSTM Model
• Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions
• Feature Fusion Effects of Tensor Product Representation on (De)Compositional Network for Caption Generation for Images
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
• ExGate: Externally Controlled Gating for Feature-based Attention in Artificial Neural Networks
• Recurrent Multimodal Interaction for Referring Image Segmentation
• Scene-based Factored Attention for Image Captioning
• PR Product: A Substitute for Inner Product in Neural Networks