Are we pretraining it right? Digging deeper into visio-linguistic pretraining

Abstract

Numerous recent works have proposed pretraining generic visio-linguistic representations and then finetuning them for downstream vision and language tasks. While architecture and objective function design choices have received attention, the choice of pretraining datasets has received little attention. In this work, we question some of the default choices made in literature. For instance, we systematically study how varying similarity between the pretraining dataset domain (textual and visual) and the downstream domain affects performance. Surprisingly, we show that automatically generated data in a domain closer to the downstream task (e.g., VQA v2) is a better choice for pretraining than "natural" data but of a slightly different domain (e.g., Conceptual Captions). On the other hand, some seemingly reasonable choices of pretraining datasets were found to be entirely ineffective for some downstream tasks. This suggests that despite the numerous recent efforts, vision & language pretraining does not quite work "out of the box" yet. Overall, as a by-product of our study, we find that simple design choices in pretraining can help us achieve close to state-of-art results on downstream tasks without any architectural changes.

23 pages, 6 figures. First two authors contributed equally. More info at https://github.com/facebookresearch/pythia

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Microsoft COCO Captions: Data Collection and Evaluation Server
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• A large annotated corpus for learning natural language inference
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• UNITER: UNiversal Image-TExt Representation Learning
• Supervised Multimodal Bitransformers for Classifying Images and Text
• Improvements to context based self-supervised learning
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• 12-in-1: Multi-Task Vision and Language Representation Learning
• On the adequacy of untuned warmup for adaptive optimization
• Visual Entailment Task for Visually-Grounded Language Learning

Cited by in corpus (15)

• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
• Detecting Hateful Memes Using a Multimodal Deep Ensemble
• Human-Adversarial Visual Question Answering
• Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
• Multimodal Fusion Refiner Networks
• CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
• Probing Image-Language Transformers for Verb Understanding
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Separating Skills and Concepts for Novel Visual Question Answering
• Detecting Harmful Memes and Their Targets
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals
• Cross-Modal Retrieval Augmentation for Multi-Modal Classification
• "Wikily" Supervised Neural Translation Tailored to Cross-Lingual Tasks