12-in-1: Multi-Task Vision and Language Representation Learning

Abstract

Much of vision-and-language research focuses on a small but diverse set of independent tasks and supporting datasets often studied in isolation; however, the visually-grounded language understanding skills required for success at these tasks overlap significantly. In this work, we investigate these relationships between vision-and-language tasks by developing a large-scale, multi-task training regime. Our approach culminates in a single model on 12 datasets from four broad categories of task including visual question answering, caption-based image retrieval, grounding referring expressions, and multi-modal verification. Compared to independently trained single-task models, this represents a reduction from approximately 3 billion parameters to 270 million while simultaneously improving performance by 2.05 points on average across tasks. We use our multi-task framework to perform in-depth analysis of the effect of joint training diverse tasks. Further, we show that finetuning task-specific models from our single multi-task model can lead to further improvements, achieving performance at or above the state-of-the-art.

Jiasen Lu and Vedanuj Goswami contributed equally to this work

References in corpus (14)

• Decoupled Weight Decay Regularization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• An Overview of Multi-Task Learning in Deep Neural Networks
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Cross-lingual Language Model Pretraining
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• UNITER: UNiversal Image-TExt Representation Learning
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• OmniNet: A unified architecture for multi-modal multi-task learning
• Visual Entailment Task for Visually-Grounded Language Learning

Cited by in corpus (4)

• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
• Visual Question Answering based on Local-Scene-Aware Referring Expression Generation
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning