Auxiliary Learning by Implicit Differentiation

Abstract

Training neural networks with auxiliary tasks is a common practice for improving the performance on a main task of interest. Two main challenges arise in this multi-task learning setting: (i) designing useful auxiliary tasks; and (ii) combining auxiliary tasks into a single coherent loss. Here, we propose a novel framework, AuxiLearn, that targets both challenges based on implicit differentiation. First, when useful auxiliaries are known, we propose learning a network that combines all losses into a single coherent objective function. This network can learn non-linear interactions between tasks. Second, when no useful auxiliary task is known, we describe how to learn a network that generates a meaningful, novel auxiliary task. We evaluate AuxiLearn in a series of tasks and domains, including image segmentation and learning with attributes in the low data regime, and find that it consistently outperforms competing methods.

arXiv commentPublished at ICLR 2021

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Survey on Multi-Task Learning
• Indoor Semantic Segmentation using depth information
• Multi-Task Learning as Multi-Objective Optimization
• Meta-Learning with Implicit Gradients
• Self-Supervised Generalisation with Meta Auxiliary Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Self-Supervised Learning for Domain Adaptation on Point-Clouds

Cited by in corpus (3)

• Learning the Pareto Front with Hypernetworks
• Teaching with Commentaries
• Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative