Contrastive Visual-Linguistic Pretraining

Abstract

Several multi-modality representation learning approaches such as LXMERT and ViLBERT have been proposed recently. Such approaches can achieve superior performance due to the high-level semantic information captured during large-scale multimodal pretraining. However, as ViLBERT and LXMERT adopt visual region regression and classification loss, they often suffer from domain gap and noisy label problems, based on the visual features having been pretrained on the Visual Genome dataset. To overcome these issues, we propose unbiased Contrastive Visual-Linguistic Pretraining (CVLP), which constructs a visual self-supervised loss built upon contrastive learning. We evaluate CVLP on several down-stream tasks, including VQA, GQA and NLVR2 to validate the superiority of contrastive learning on multi-modality representation learning. Our code is available at: https://github.com/ArcherYunDong/CVLP-.

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• A Simple Framework for Contrastive Learning of Visual Representations
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Improved Baselines with Momentum Contrastive Learning
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• Momentum Contrast for Unsupervised Visual Representation Learning
• Contrastive Multiview Coding
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Reducing Transformer Depth on Demand with Structured Dropout
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• UNITER: UNiversal Image-TExt Representation Learning
• Multi-modality Latent Interaction Network for Visual Question Answering
• Character Matters: Video Story Understanding with Character-Aware Relations

Cited by in corpus (4)

• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• Language-Driven Image Style Transfer