HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections

Abstract

Achieving state-of-the-art performance on natural language understanding tasks typically relies on fine-tuning a fresh model for every task. Consequently, this approach leads to a higher overall parameter cost, along with higher technical maintenance for serving multiple models. Learning a single multi-task model that is able to do well for all the tasks has been a challenging and yet attractive proposition. In this paper, we propose \textsc{HyperGrid}, a new approach for highly effective multi-task learning. The proposed approach is based on a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks. In order to construct the proposed hypernetwork, our method learns the interactions and composition between a global (task-agnostic) state and a local task-specific state. We apply our proposed \textsc{HyperGrid} on the current state-of-the-art T5 model, demonstrating strong performance across the GLUE and SuperGLUE benchmarks when using only a single multi-task model. Our method helps bridge the gap between fine-tuning and multi-task learning approaches.

References in corpus (16)

• Sequence to Sequence Learning with Neural Networks
• Overcoming catastrophic forgetting in neural networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• An Overview of Multi-Task Learning in Deep Neural Networks
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• The Natural Language Decathlon: Multitask Learning as Question Answering
• One Model To Learn Them All
• Multi-Task Deep Neural Networks for Natural Language Understanding
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• Get To The Point: Summarization with Pointer-Generator Networks
• A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
• Scalable and Order-robust Continual Learning with Additive Parameter Decomposition