More Grounded Image Captioning by Distilling Image-Text Matching Model
arXiv:2004.00390
Abstract
Visual attention not only improves the performance of image captioners, but also serves as a visual interpretation to qualitatively measure the caption rationality and model transparency. Specifically, we expect that a captioner can fix its attentive gaze on the correct objects while generating the corresponding words. This ability is also known as grounded image captioning. However, the grounding accuracy of existing captioners is far from satisfactory. To improve the grounding accuracy while retaining the captioning quality, it is expensive to collect the word-region alignment as strong supervision. To this end, we propose a Part-of-Speech (POS) enhanced image-text matching model (SCAN \cite{lee2018stacked}): POS-SCAN, as the effective knowledge distillation for more grounded image captioning. The benefits are two-fold: 1) given a sentence and an image, POS-SCAN can ground the objects more accurately than SCAN; 2) POS-SCAN serves as a word-region alignment regularization for the captioner's visual attention module. By showing benchmark experimental results, we demonstrate that conventional image captioners equipped with POS-SCAN can significantly improve the grounding accuracy without strong supervision. Last but not the least, we explore the indispensable Self-Critical Sequence Training (SCST) \cite{Rennie_2017_CVPR} in the context of grounded image captioning and show that the image-text matching score can serve as a reward for more grounded captioning \footnote{https://github.com/YuanEZhou/Grounded-Image-Captioning}.
arXiv commentAccepted by CVPR 2020
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Distilling the Knowledge in a Neural Network
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- FitNets: Hints for Thin Deep Nets
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Context-Aware Visual Policy Network for Fine-Grained Image Captioning
- Context-Aware Visual Policy Network for Sequence-Level Image Captioning
- Women also Snowboard: Overcoming Bias in Captioning Models
- Attention Correctness in Neural Image Captioning
- Discriminability objective for training descriptive captions
- Unbiased Scene Graph Generation from Biased Training
- Joint Visual Grounding with Language Scene Graphs