LAMP: Label Augmented Multimodal Pretraining
arXiv:2012.04446
Abstract
Multi-modal representation learning by pretraining has become an increasing interest due to its easy-to-use and potential benefit for various Visual-and-Language~(V-L) tasks. However its requirement of large volume and high-quality vision-language pairs highly hinders its values in practice. In this paper, we proposed a novel label-augmented V-L pretraining model, named LAMP, to address this problem. Specifically, we leveraged auto-generated labels of visual objects to enrich vision-language pairs with fine-grained alignment and correspondingly designed a novel pretraining task. Besides, we also found such label augmentation in second-stage pretraining would further universally benefit various downstream tasks. To evaluate LAMP, we compared it with some state-of-the-art models on four downstream tasks. The quantitative results and analysis have well proven the value of labels in V-L pretraining and the effectiveness of LAMP.
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Microsoft COCO Captions: Data Collection and Evaluation Server
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models