Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present
arXiv:1803.11439
Abstract
Recently, caption generation with an encoder-decoder framework has been extensively studied and applied in different domains, such as image captioning, code captioning, and so on. In this paper, we propose a novel architecture, namely Auto-Reconstructor Network (ARNet), which, coupling with the conventional encoder-decoder framework, works in an end-to-end fashion to generate captions. ARNet aims at reconstructing the previous hidden state with the present one, besides behaving as the input-dependent transition operator. Therefore, ARNet encourages the current hidden state to embed more information from the previous one, which can help regularize the transition dynamics of recurrent neural networks (RNNs). Extensive experimental results show that our proposed ARNet boosts the performance over the existing encoder-decoder models on both image captioning and source code captioning tasks. Additionally, ARNet remarkably reduces the discrepancy between training and inference processes for caption generation. Furthermore, the performance on permuted sequential MNIST demonstrates that ARNet can effectively regularize RNN, especially on modeling long-term dependencies. Our code is available at: https://github.com/chenxinpeng/ARNet
Accepted by CVPR 2018
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Recurrent Neural Network Regularization
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Going Deeper with Convolutions
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Rethinking the Inception Architecture for Computer Vision
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Image Captioning with Semantic Attention
- How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?
- Show and Tell: A Neural Image Caption Generator
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Review Networks for Caption Generation
- CIDEr: Consensus-based Image Description Evaluation
- SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
- Boosting Image Captioning with Attributes