BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

Abstract

Multi-task learning shares information between related tasks, sometimes reducing the number of parameters required. State-of-the-art results across multiple natural language understanding tasks in the GLUE benchmark have previously used transfer from a single large task: unsupervised pre-training with BERT, where a separate BERT model was fine-tuned for each task. We explore multi-task approaches that share a single BERT model with a small number of additional task-specific parameters. Using new adaptation modules, PALs or `projected attention layers', we match the performance of separately fine-tuned models on the GLUE benchmark with roughly 7 times fewer parameters, and obtain state-of-the-art results on the Recognizing Textual Entailment dataset.

Accepted for publication at ICML 2019

Cited by in corpus (35)

• Pre-trained Models for Natural Language Processing: A Survey
• Revisiting Few-sample BERT Fine-tuning
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Parameter-Efficient Transfer Learning for NLP
• Multi-task learning for natural language processing in the 2020s: where are we going?
• A Survey on Transfer Learning in Natural Language Processing
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
• Multi-Task Learning in Natural Language Processing: An Overview
• Continual Learning with Adaptive Weights (CLAW)
• Parameter-Efficient Transfer Learning with Diff Pruning
• Deep Transformers with Latent Depth
• An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers
• To Tune or Not To Tune? How About the Best of Both Worlds?
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
• Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Contextual Lensing of Universal Sentence Representations
• Few-shot Sequence Learning with Transformers
• Knowledge Distillation for Multi-task Learning
• HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
• Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
• Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining
• Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
• MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
• Deep Anomaly Detection by Residual Adaptation
• Knowledge Transfer via Pre-training for Recommendation: A Review and Prospect
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model
• Neural Architecture Search for Sentence Classification with BERT
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization