Regularizing Deep Multi-Task Networks using Orthogonal Gradients

Abstract

Deep neural networks are a promising approach towards multi-task learning because of their capability to leverage knowledge across domains and learn general purpose representations. Nevertheless, they can fail to live up to these promises as tasks often compete for a model's limited resources, potentially leading to lower overall performance. In this work we tackle the issue of interfering tasks through a comprehensive analysis of their training, derived from looking at the interaction between gradients within their shared parameters. Our empirical results show that well-performing models have low variance in the angles between task gradients and that popular regularization methods implicitly reduce this measure. Based on this observation, we propose a novel gradient regularization term that minimizes task interference by enforcing near orthogonal gradients. Updating the shared parameters using this property encourages task specific decoders to optimize different parts of the feature extractor, thus reducing competition. We evaluate our method with classification and regression tasks on the multiDigitMNIST, NYUv2 and SUN RGB-D datasets where we obtain competitive results.

11 pages, 5 figures

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• How transferable are features in deep neural networks?
• An Overview of Multi-Task Learning in Deep Neural Networks
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Multi-Task Learning as Multi-Objective Optimization
• Learning Multiple Tasks with Multilinear Relationship Networks
• Adapting Auxiliary Losses Using Gradient Similarity
• Regularizing CNNs with Locally Constrained Decorrelations
• Trace Norm Regularised Deep Multi-Task Learning
• Fast Scene Understanding for Autonomous Driving
• Meta-Learning Representations for Continual Learning

Cited by in corpus (13)

• Multi-Task Learning for Dense Prediction Tasks: A Survey
• Gradient Surgery for Multi-Task Learning
• Detecting 32 Pedestrian Attributes for Autonomous Vehicles
• Efficiently Identifying Task Groupings for Multi-Task Learning
• Multi-Task Reinforcement Learning with Context-based Representations
• Estimating and Improving Fairness with Adversarial Learning
• Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery
• Adaptive Sparse Transformer for Multilingual Translation
• Measuring and Harnessing Transference in Multi-Task Learning
• Robust Optimization for Multilingual Translation with Imbalanced Data
• GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning
• Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral
• Disentanglement Learning via Topology