RotoGrad: Gradient Homogenization in Multitask Learning
arXiv:2103.02631
Abstract
Multitask learning is being increasingly adopted in applications domains like computer vision and reinforcement learning. However, optimally exploiting its advantages remains a major challenge due to the effect of negative transfer. Previous works have tracked down this issue to the disparities in gradient magnitudes and directions across tasks, when optimizing the shared network parameters. While recent work has acknowledged that negative transfer is a two-fold problem, existing approaches fall short as they only focus on either homogenizing the gradient magnitude across tasks; or greedily change the gradient directions, overlooking future conflicts. In this work, we introduce RotoGrad, an algorithm that tackles negative transfer as a whole: it jointly homogenizes gradient magnitudes and directions, while ensuring training convergence. We show that RotoGrad outperforms competing methods in complex problems, including multi-label classification in CelebA and computer vision tasks in the NYUv2 dataset. A Pytorch implementation can be found in https://github.com/adrianjav/rotograd.
arXiv commentSpotlight at ICLR 2022. 24 pages, 9 figures
References in corpus (10)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- An Overview of Multi-Task Learning in Deep Neural Networks
- Indoor Semantic Segmentation using depth information
- Multi-Task Learning as Multi-Objective Optimization
- Which Tasks Should Be Learned Together in Multi-task Learning?
- AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
- Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
- Meta-Learning with Warped Gradient Descent
- Convergence of Learning Dynamics in Stackelberg Games
- Branched Multi-Task Networks: Deciding What Layers To Share