Unifying Vision-and-Language Tasks via Text Generation

Abstract

Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual question answering, a region scorer for referring expression comprehension, and a language decoder for image captioning, etc. To alleviate these hassles, in this work, we propose a unified framework that learns different tasks in a single architecture with the same language modeling objective, i.e., multimodal conditional text generation, where our models learn to generate labels in text based on the visual and textual inputs. On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models. Moreover, our generative approach shows better generalization ability on questions that have rare answers. Also, we show that our framework allows multi-task learning in a single architecture with a single set of parameters, achieving similar performance to separately optimized single-task models. Our code is publicly available at: https://github.com/j-min/VL-T5

arXiv commentICML 2021 (15 pages, 4 figures, 14 tables)

References in corpus (14)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Decoupled Weight Decay Regularization
• Learning Transferable Visual Models From Natural Language Supervision
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Mixed Precision Training
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Towards Automatic Learning of Procedures from Web Instructional Videos
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Making Pre-trained Language Models Better Few-shot Learners
• MAttNet: Modular Attention Network for Referring Expression Comprehension
• Modeling Context in Referring Expressions
• XGPT: Cross-modal Generative Pre-Training for Image Captioning

Cited by in corpus (15)

• On the Opportunities and Risks of Foundation Models
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• VLP: A Survey on Vision-Language Pre-training
• Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
• Rethinking Search: Making Domain Experts out of Dilettantes
• Multimodal Few-Shot Learning with Frozen Language Models
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
• UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• Scaling Up Vision-Language Pre-training for Image Captioning
• Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• Towards General Purpose Vision Systems
• Less is More: Generating Grounded Navigation Instructions from Landmarks