An Empirical Study of Training End-to-End Vision-and-Language Transformers

Abstract

Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks. While recent work has shown that fully transformer-based VL models can be more efficient than previous region-feature-based methods, their performance on downstream tasks often degrades significantly. In this paper, we present METER, a Multimodal End-to-end TransformER framework, through which we investigate how to design and pre-train a fully transformer-based VL model in an end-to-end manner. Specifically, we dissect the model designs along multiple dimensions: vision encoders (e.g., CLIP-ViT, Swin transformer), text encoders (e.g., RoBERTa, DeBERTa), multimodal fusion module (e.g., merged attention vs. co-attention), architectural design (e.g., encoder-only vs. encoder-decoder), and pre-training objectives (e.g., masked image modeling). We conduct comprehensive experiments and provide insights on how to train a performant VL transformer. METER achieves an accuracy of 77.64% on the VQAv2 test-std set using only 4M images for pre-training, surpassing the state-of-the-art region-feature-based model by 1.04%, and outperforming the previous best fully transformer-based model by 1.6%. Notably, when further scaled up, our best VQA model achieves an accuracy of 80.54%. Code and pre-trained models are released at https://github.com/zdou0830/METER.

CVPR 2022

References in corpus (15)

• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Zero-Shot Text-to-Image Generation
• BEiT: BERT Pre-Training of Image Transformers
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Florence: A New Foundation Model for Computer Vision
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Unifying Vision-and-Language Tasks via Text Generation
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• VOLO: Vision Outlooker for Visual Recognition

Cited by in corpus (1)

• Florence: A New Foundation Model for Computer Vision