Adapting Auxiliary Losses Using Gradient Similarity

Abstract

One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations. However, it is not always trivial to know if an auxiliary task will be helpful for the main task and when it could start hurting. We propose to use the cosine similarity between gradients of tasks as an adaptive weight to detect when an auxiliary loss is helpful to the main loss. We show that our approach is guaranteed to converge to critical points of the main task and demonstrate the practical usefulness of the proposed algorithm in a few domains: multi-task supervised learning on subsets of ImageNet, reinforcement learning on gridworld, and reinforcement learning on Atari games.

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• How transferable are features in deep neural networks?
• Opening the Black Box of Deep Neural Networks via Information
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Learning to Navigate in Complex Environments
• Multi-Task Learning as Multi-Objective Optimization
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Essentially No Barriers in Neural Network Energy Landscape
• Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification
• Kickstarting Deep Reinforcement Learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Multi-task Deep Reinforcement Learning with PopArt
• Agent Modeling as Auxiliary Task for Deep Reinforcement Learning
• Deep Reinforcement Learning for Doom using Unsupervised Auxiliary Tasks

Cited by in corpus (33)

• A Survey and Critique of Multiagent Deep Reinforcement Learning
• Gradient Surgery for Multi-Task Learning
• Improving Adversarial Robustness of Ensembles with Diversity Training
• Self-Supervised Generalisation with Meta Auxiliary Learning
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Learning explanations that are hard to vary
• Multi-Task Reinforcement Learning with Soft Modularization
• Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
• MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Regularizing Deep Multi-Task Networks using Orthogonal Gradients
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• Multi-Task Reinforcement Learning with Context-based Representations
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Semi-Supervised Learning by Label Gradient Alignment
• Auxiliary Task Reweighting for Minimum-data Learning
• Generalization in multitask deep neural classifiers: a statistical physics approach
• Multi-Task Learning in Histo-pathology for Widely Generalizable Model
• A Brief Review of Deep Multi-task Learning and Auxiliary Task Learning
• Knowledge Distillation for Multi-task Learning
• GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning
• Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral
• Optimizing Data Usage via Differentiable Rewards
• Auxiliary Learning by Implicit Differentiation
• ImageNet-21K Pretraining for the Masses
• On The Effect of Auxiliary Tasks on Representation Dynamics
• Using dynamic loss weighting to boost improvements in forecast stability
• Multiple Instance Learning with Auxiliary Task Weighting for Multiple Myeloma Classification
• Towards a practical measure of interference for reinforcement learning
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks
• Improving RNN Transducer Based ASR with Auxiliary Tasks
• Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction
• Adaptive Transfer Learning on Graph Neural Networks