Auxiliary Tasks in Multi-task Learning

Abstract

Multi-task convolutional neural networks (CNNs) have shown impressive results for certain combinations of tasks, such as single-image depth estimation (SIDE) and semantic segmentation. This is achieved by pushing the network towards learning a robust representation that generalizes well to different atomic tasks. We extend this concept by adding auxiliary tasks, which are of minor relevance for the application, to the set of learned tasks. As a kind of additional regularization, they are expected to boost the performance of the ultimately desired main tasks. To study the proposed approach, we picked vision-based road scene understanding (RSU) as an exemplary application. Since multi-task learning requires specialized datasets, particularly when using extensive sets of tasks, we provide a multi-modal dataset for multi-task RSU, called synMT. More than 2.5 $\cdot$ 10^5 synthetic images, annotated with 21 different labels, were acquired from the video game Grand Theft Auto V (GTA V). Our proposed deep multi-task CNN architecture was trained on various combination of tasks using synMT. The experiments confirmed that auxiliary tasks can indeed boost network performance, both in terms of final results and training time.

arXiv commentfixed minor typesetting issue

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Rethinking Atrous Convolution for Semantic Image Segmentation
• An Overview of Multi-Task Learning in Deep Neural Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications
• Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications

Cited by in corpus (22)

• Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarios
• Self-Supervised Generalisation with Meta Auxiliary Learning
• Faster Training of Mask R-CNN by Focusing on Instance Boundaries
• GlassNet: a multitask deep neural network for predicting many glass properties
• A Multi-modal Fusion Framework Based on Multi-task Correlation Learning for Cancer Prognosis Prediction
• MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks
• AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving
• Multi-Modal Multi-Task (3MT) Road Segmentation
• Enhancing User Behavior Sequence Modeling by Generative Tasks for Session Search
• Constructing Time-Series Momentum Portfolios with Deep Multi-Task Learning
• To Balance or Not to Balance: A Simple-yet-Effective Approach for Learning with Long-Tailed Distributions
• Multitask Learning in Minimally Invasive Surgical Vision: A Review
• Learning Discriminative features using Center Loss and Reconstruction as Regularizer for Speech Emotion Recognition
• SemSegDepth: A Combined Model for Semantic Segmentation and Depth Completion
• Stratospheric Aerosol Injection as a Deep Reinforcement Learning Problem
• Adaptive Weight Assignment Scheme For Multi-task Learning
• Supervising the Decoder of Variational Autoencoders to Improve Scientific Utility
• SALT: Subspace Alignment as an Auxiliary Learning Task for Domain Adaptation
• CubiCasa5K: A Dataset and an Improved Multi-Task Model for Floorplan Image Analysis
• Disentangling Transfer and Interference in Multi-Domain Learning
• Learning to Generalize One Sample at a Time with Self-Supervision
• Fourier Transform Approximation as an Auxiliary Task for Image Classification