Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models

Abstract

Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization. While it is a common practice to apply a language-agnostic procedure optimizing a joint multilingual task objective, how to properly characterize and take advantage of its underlying problem structure for improving optimization efficiency remains under-explored. In this paper, we attempt to peek into the black-box of multilingual optimization through the lens of loss function geometry. We find that gradient similarity measured along the optimization trajectory is an important signal, which correlates well with not only language proximity but also the overall model performance. Such observation helps us to identify a critical limitation of existing gradient-based multi-task learning methods, and thus we derive a simple and scalable optimization procedure, named Gradient Vaccine, which encourages more geometrically aligned parameter updates for close tasks. Empirically, our method obtains significant model performance gains on multilingual machine translation and XTREME benchmark tasks for multilingual language models. Our work reveals the importance of properly measuring and utilizing language proximity in multilingual optimization, and has broader implications for multi-task learning beyond multilingual modeling.

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• An Overview of Multi-Task Learning in Deep Neural Networks
• Cross-lingual Language Model Pretraining
• On the Cross-lingual Transferability of Monolingual Representations
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Multilingual Alignment of Contextual Word Representations
• How multilingual is Multilingual BERT?
• A Convex Formulation for Learning Task Relationships in Multi-Task Learning
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• Gradient Surgery for Multi-Task Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
• Pareto Multi-Task Learning
• Routing Networks and the Challenges of Modular and Compositional Computation
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Multilingual NMT with a language-independent attention bridge
• Adaptive Scheduling for Multi-Task Learning
• Emerging Cross-lingual Structure in Pretrained Language Models
• Balancing Training for Multilingual Neural Machine Translation
• Towards more Reliable Transfer Learning

Cited by in corpus (12)

• Transformer-based Multimodal Change Detection with Multitask Consistency Constraints
• Multi-Task Learning in Natural Language Processing: An Overview
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• Scaling End-to-End Models for Large-Scale Multilingual ASR
• Adaptive Sparse Transformer for Multilingual Translation
• Multitask Learning in Minimally Invasive Surgical Vision: A Review
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
• Measuring and Harnessing Transference in Multi-Task Learning
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Multi-Head Adapter Routing for Cross-Task Generalization
• Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning
• Improving Multilingual Translation by Representation and Gradient Regularization