HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks

Abstract

Multi-task learning (MTL) can improve performance on a task by sharing representations with one or more related auxiliary-tasks. Usually, MTL-networks are trained on a composite loss function formed by a constant weighted combination of the separate task losses. In practice, constant loss weights lead to poor results for two reasons: (i) the relevance of the auxiliary tasks can gradually drift throughout the learning process; (ii) for mini-batch based optimisation, the optimal task weights vary significantly from one update to the next depending on mini-batch sample composition. We introduce HydaLearn, an intelligent weighting algorithm that connects main-task gain to the individual task gradients, in order to inform dynamic loss weighting at the mini-batch level, addressing i and ii. Using HydaLearn, we report performance increases on synthetic data, as well as on two supervised learning domains.

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• An overview of gradient descent optimization algorithms
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Multi-Task Learning as Multi-Objective Optimization
• An Empirical Model of Large-Batch Training
• Gradient Surgery for Multi-Task Learning
• The Benefit of Multitask Representation Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• Benchmark of Deep Learning Models on Large Healthcare MIMIC Datasets
• Theory of Deep Learning IIb: Optimization Properties of SGD
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Adaptive Scheduling for Multi-Task Learning
• Identifying beneficial task relations for multi-task learning in deep neural networks
• AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning