Exceeding the Limits of Visual-Linguistic Multi-Task Learning
arXiv:2107.13054
Abstract
By leveraging large amounts of product data collected across hundreds of live e-commerce websites, we construct 1000 unique classification tasks that share similarly-structured input data, comprised of both text and images. These classification tasks focus on learning the product hierarchy of different e-commerce websites, causing many of them to be correlated. Adopting a multi-modal transformer model, we solve these tasks in unison using multi-task learning (MTL). Extensive experiments are presented over an initial 100-task dataset to reveal best practices for "large-scale MTL" (i.e., MTL with more than 100 tasks). From these experiments, a final, unified methodology is derived, which is composed of both best practices and new proposals such as DyPa, a simple heuristic for automatically allocating task-specific parameters to tasks that could benefit from extra capacity. Using our large-scale MTL methodology, we successfully train a single model across all 1000 tasks in our dataset while using minimal task specific parameters, thereby showing that it is possible to extend several orders of magnitude beyond current efforts in MTL.
arXiv comment10 pages, 7 figures
References in corpus (38)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Language Models are Few-Shot Learners
- An Overview of Multi-Task Learning in Deep Neural Networks
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Linformer: Self-Attention with Linear Complexity
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Reducing Transformer Depth on Demand with Structured Dropout
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- Multi-Task Deep Neural Networks for Natural Language Understanding
- UNITER: UNiversal Image-TExt Representation Learning
- Supervised Multimodal Bitransformers for Classifying Images and Text
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
- VideoBERT: A Joint Model for Video and Language Representation Learning
- Parameter-Efficient Transfer Learning for NLP
- ERNIE: Enhanced Language Representation with Informative Entities
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- The Cost of Training NLP Models: A Concise Overview
- BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
- Gradient Surgery for Multi-Task Learning
- Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- Multi-task learning for natural language processing in the 2020s: where are we going?
- Unified Vision-Language Pre-Training for Image Captioning and VQA
- BERT Loses Patience: Fast and Robust Inference with Early Exit
- 12-in-1: Multi-Task Vision and Language Representation Learning
- Are we pretraining it right? Digging deeper into visio-linguistic pretraining
- End-to-End Multi-Task Learning with Attention
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
- OmniNet: A unified architecture for multi-modal multi-task learning
- Small Towers Make Big Differences
- Deep Asymmetric Multi-task Feature Learning