Auxiliary Learning for Deep Multi-task Learning

Abstract

Multi-task learning (MTL) is an efficient solution to solve multiple tasks simultaneously in order to get better speed and performance than handling each single-task in turn. The most current methods can be categorized as either: (i) hard parameter sharing where a subset of the parameters is shared among tasks while other parameters are task-specific; or (ii) soft parameter sharing where all parameters are task-specific but they are jointly regularized. Both methods suffer from limitations: the shared hidden layers of the former are difficult to optimize due to the competing objectives while the complexity of the latter grows linearly with the increasing number of tasks. To mitigate those drawbacks, this paper proposes an alternative, where we explicitly construct an auxiliary module to mimic the soft parameter sharing for assisting the optimization of the hard parameter sharing layers in the training phase. In particular, the auxiliary module takes the outputs of the shared hidden layers as inputs and is supervised by the auxiliary task loss. During training, the auxiliary module is jointly optimized with the MTL network, serving as a regularization by introducing an inductive bias to the shared layers. In the testing phase, only the original MTL network is kept. Thus our method avoids the limitation of both categories. We evaluate the proposed auxiliary module on pixel-wise prediction tasks, including semantic segmentation, depth estimation, and surface normal prediction with different network structures. The extensive experiments over various settings verify the effectiveness of our methods.

arXiv commentFirst two authors contributed equally to this work

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• Rethinking the Value of Network Pruning
• Fully Convolutional Networks for Semantic Segmentation
• A Convergence Theory for Deep Learning via Over-Parameterization
• Pruning Filters for Efficient ConvNets
• OCNet: Object Context Network for Scene Parsing
• Multi-Task Learning as Multi-Objective Optimization
• Efficient Architecture Search by Network Transformation
• Learning Multiple Tasks with Multilinear Relationship Networks
• Training Deeper Convolutional Networks with Deep Supervision
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Light-Weight RefineNet for Real-Time Semantic Segmentation
• Deep Multi-task Representation Learning: A Tensor Factorisation Approach
• Trace Norm Regularised Deep Multi-Task Learning
• Deep attention-based classification network for robust depth prediction

Cited by in corpus (4)

• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Switchable Precision Neural Networks
• A Brief Review of Deep Multi-task Learning and Auxiliary Task Learning
• Learning task-agnostic representation via toddler-inspired learning