Gradient Surgery for Multi-Task Learning

Abstract

While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimization challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single-task learning are not fully understood. In this work, we identify a set of three conditions of the multi-task optimization landscape that cause detrimental gradient interference, and develop a simple yet general approach for avoiding such interference between task gradients. We propose a form of gradient surgery that projects a task's gradient onto the normal plane of the gradient of any other task that has a conflicting gradient. On a series of challenging multi-task supervised and multi-task RL problems, this approach leads to substantial gains in efficiency and performance. Further, it is model-agnostic and can be combined with previously-proposed multi-task architectures for enhanced performance.

arXiv commentNeurIPS 2020. Code is available at https://github.com/tianheyu927/PCGrad

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• An Overview of Multi-Task Learning in Deep Neural Networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Multi-Task Learning as Multi-Objective Optimization
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Qualitatively characterizing neural network optimization problems
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• Policy Distillation
• Trace Norm Regularised Deep Multi-Task Learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Routing Networks and the Challenges of Modular and Compositional Computation
• Orthogonal Gradient Descent for Continual Learning
• Regularizing Deep Multi-Task Networks using Orthogonal Gradients
• End-to-End Multi-Task Learning with Attention

Cited by in corpus (52)

• Multi-Task Learning with Deep Neural Networks: A Survey
• Physics-informed neural networks for the shallow-water equations on the sphere
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Multi-Task Reinforcement Learning with Soft Modularization
• On Warm-Starting Neural Network Training
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• La-MAML: Look-ahead Meta Learning for Continual Learning
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Efficiently Identifying Task Groupings for Multi-Task Learning
• Improved Schemes for Episodic Memory-based Lifelong Learning
• Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning
• Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control
• FedH2L: Federated Learning with Model and Statistical Heterogeneity
• Multi-Task Reinforcement Learning with Context-based Representations
• Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning
• Multitask Learning Strengthens Adversarial Robustness
• Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent
• Multi-Task Learning in Histo-pathology for Widely Generalizable Model
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
• Measuring and Harnessing Transference in Multi-Task Learning
• SpeechNet: A Universal Modularized Model for Speech Processing Tasks
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithms
• Small Towers Make Big Differences
• Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral
• Knowledge Distillation for Multi-task Learning
• Multi-Task Self-Training for Learning General Representations
• ImageNet-21K Pretraining for the Masses
• Multi-Objective Learning to Predict Pareto Fronts Using Hypervolume Maximization
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Federated Learning with Fair Averaging
• SimPose: Effectively Learning DensePose and Surface Normals of People from Simulated Data
• Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learning
• CheXseen: Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays
• Multi-modal Residual Perceptron Network for Audio-Video Emotion Recognition
• Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks
• Multi-Task Neural Processes
• Unbiased Sentence Encoder For Large-Scale Multi-lingual Search Engines
• Suboptimal coverings for continuous spaces of control tasks
• Autonomous Reinforcement Learning via Subgoal Curricula
• Lipschitz standardization for multivariate learning
• Fast Line Search for Multi-Task Learning
• Mix and Mask Actor-Critic Methods
• Meta-Learning to Improve Pre-Training
• Improving Multilingual Translation by Representation and Gradient Regularization
• Supervised Seeded Iterated Learning for Interactive Language Learning