VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning

Abstract

It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated in the novel object captioning challenge (nocaps). In this challenge, no additional image-caption training data, other thanCOCO Captions, is allowed for model training. Thus, conventional Vision-Language Pre-training (VLP) methods cannot be applied. This paper presents VIsual VOcabulary pretraining (VIVO) that performs pre-training in the absence of caption annotations. By breaking the dependency of paired image-caption training data in VLP, VIVO can leverage large amounts of paired image-tag data to learn a visual vocabulary. This is done by pre-training a multi-layer Transformer model that learns to align image-level tags with their corresponding image region features. To address the unordered nature of image tags, VIVO uses a Hungarian matching loss with masked tag prediction to conduct pre-training. We validate the effectiveness of VIVO by fine-tuning the pre-trained model for image captioning. In addition, we perform an analysis of the visual-text alignment inferred by our model. The results show that our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects. Our single model has achieved new state-of-the-art results on nocaps and surpassed the human CIDEr score.

arXiv commentAAAI 2021

References in corpus (4)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Decoupled Novel Object Captioner

Cited by in corpus (8)

• Deep Learning Approaches on Image Captioning: A Review
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• A Thorough Review on Recent Deep Learning Methodologies for Image Captioning
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
• Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
• Is Object Detection Necessary for Human-Object Interaction Recognition?
• Maria: A Visual Experience Powered Conversational Agent
• Partially-Supervised Novel Object Captioning Leveraging Context from Paired Data