Hidden State Guidance: Improving Image Captioning using An Image Conditioned Autoencoder

Abstract

Most RNN-based image captioning models receive supervision on the output words to mimic human captions. Therefore, the hidden states can only receive noisy gradient signals via layers of back-propagation through time, leading to less accurate generated captions. Consequently, we propose a novel framework, Hidden State Guidance (HSG), that matches the hidden states in the caption decoder to those in a teacher decoder trained on an easier task of autoencoding the captions conditioned on the image. During training with the REINFORCE algorithm, the conventional rewards are sentence-based evaluation metrics equally distributed to each generated word, no matter their relevance. HSG provides a word-level reward that helps the model learn better hidden representations. Experimental results demonstrate that HSG clearly outperforms various state-of-the-art caption decoders using either raw images or detected objects as inputs.

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• FitNets: Hints for Thin Deep Nets
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Sequence-Level Knowledge Distillation
• Discriminability objective for training descriptive captions
• Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
• Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data
• Auto-Encoding Scene Graphs for Image Captioning
• Faithful Multimodal Explanation for Visual Question Answering
• Generating Question Relevant Captions to Aid Visual Question Answering