Adaptive Scheduling for Multi-Task Learning

Abstract

To train neural machine translation models simultaneously on multiple tasks (languages), it is common to sample each task uniformly or in proportion to dataset sizes. As these methods offer little control over performance trade-offs, we explore different task scheduling approaches. We first consider existing non-adaptive techniques, then move on to adaptive schedules that over-sample tasks with poorer results compared to their respective baseline. As explicit schedules can be inefficient, especially if one task is highly over-sampled, we also consider implicit schedules, learning to scale learning rates or gradients of individual tasks instead. These techniques allow training multilingual models that perform better for low-resource language pairs (tasks with small amount of data), while minimizing negative effects on high-resource tasks.

arXiv commentContinual Learning Workshop at NeurIPS 2018

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Online Learning Rate Adaptation with Hypergradient Descent
• Automated Curriculum Learning for Neural Networks
• AutoLoss: Learning Discrete Schedules for Alternate Optimization
• End-to-End Multi-Task Learning with Attention

Cited by in corpus (10)

• Multi-Task Learning with Deep Neural Networks: A Survey
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• A Comprehensive Survey of Multilingual Neural Machine Translation
• Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks
• Competence-based Curriculum Learning for Multilingual Machine Translation
• Balancing Average and Worst-case Accuracy in Multitask Learning