BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

Abstract

Multi-task learning shares information between related tasks, sometimes reducing the number of parameters required. State-of-the-art results across multiple natural language understanding tasks in the GLUE benchmark have previously used transfer from a single large task: unsupervised pre-training with BERT, where a separate BERT model was fine-tuned for each task. We explore multi-task approaches that share a single BERT model with a small number of additional task-specific parameters. Using new adaptation modules, PALs or `projected attention layers', we match the performance of separately fine-tuned models on the GLUE benchmark with roughly 7 times fewer parameters, and obtain state-of-the-art results on the Recognizing Textual Entailment dataset.

Accepted for publication at ICML 2019

Cited by in corpus (20)

• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Parameter-Efficient Transfer Learning for NLP
• Multi-task learning for natural language processing in the 2020s: where are we going?
• A Survey on Transfer Learning in Natural Language Processing
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• To Tune or Not To Tune? How About the Best of Both Worlds?
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
• ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
• Contextual Lensing of Universal Sentence Representations
• Knowledge Distillation for Multi-task Learning
• Few-shot Sequence Learning with Transformers
• HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
• MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
• Knowledge Transfer via Pre-training for Recommendation: A Review and Prospect
• Deep Anomaly Detection by Residual Adaptation
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization
• MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model