Multi-Task Learning for Dense Prediction Tasks: A Survey

Abstract

With the advent of deep learning, many dense prediction tasks, i.e. tasks that produce pixel-level predictions, have seen significant performance improvements. The typical approach is to learn these tasks in isolation, that is, a separate neural network is trained for each individual task. Yet, recent multi-task learning (MTL) techniques have shown promising results w.r.t. performance, computations and/or memory footprint, by jointly tackling multiple tasks through a learned shared representation. In this survey, we provide a well-rounded view on state-of-the-art deep learning approaches for MTL in computer vision, explicitly emphasizing on dense prediction tasks. Our contributions concern the following. First, we consider MTL from a network architecture point-of-view. We include an extensive overview and discuss the advantages/disadvantages of recent popular MTL models. Second, we examine various optimization methods to tackle the joint learning of multiple tasks. We summarize the qualitative elements of these works and explore their commonalities and differences. Finally, we provide an extensive experimental evaluation across a variety of dense prediction benchmarks to examine the pros and cons of the different methods, including both architectural and optimization based strategies.

Accepted to T-PAMI. Code + Suppl. Mat. can be found here: https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch IEEE Copyright Notice

References in corpus (31)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Neural Architecture Search with Reinforcement Learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• How transferable are features in deep neural networks?
• An Overview of Multi-Task Learning in Deep Neural Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• A Survey on Multi-Task Learning
• Learning multiple visual domains with residual adapters
• Multi-Task Feature Learning Via Efficient l2,1-Norm Minimization
• OCNet: Object Context Network for Scene Parsing
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Multi-Task Learning as Multi-Objective Optimization
• Clustered Multi-Task Learning: A Convex Formulation
• Learning Task Grouping and Overlap in Multi-task Learning
• Which Tasks Should Be Learned Together in Multi-task Learning?
• Learning Multiple Tasks with Multilinear Relationship Networks
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• PixelNet: Representation of the pixels, by the pixels, and for the pixels
• Pareto Multi-Task Learning
• Fast Scene Understanding for Autonomous Driving
• Bayesian Multitask Learning with Latent Hierarchies
• Regularizing Deep Multi-Task Networks using Orthogonal Gradients
• Gradient Adversarial Training of Neural Networks
• Multitask Learning Strengthens Adversarial Robustness
• Feature Partitioning for Efficient Multi-Task Architectures
• Branched Multi-Task Networks: Deciding What Layers To Share

Cited by in corpus (49)

• FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking
• MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification
• OGMN: Occlusion-guided Multi-task Network for Object Detection in UAV Images
• Enriched physics-informed neural networks for in-plane crack problems: Theory and MATLAB codes
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Survey
• Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments
• Attention-Enhanced Cross-Task Network for Analysing Multiple Attributes of Lung Nodules in CT
• MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks
• Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey
• Concurrent ischemic lesion age estimation and segmentation of CT brain using a Transformer-based network
• Sharing to learn and learning to share; Fitting together Meta-Learning, Multi-Task Learning, and Transfer Learning: A meta review
• Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings
• Controllable Pareto Multi-Task Learning
• Cyclic Learning: Bridging Image-level Labels and Nuclei Instance Segmentation
• A Multi-task Learning Framework for Product Ranking with BERT
• Efficient Multi-Task Scene Analysis with RGB-D Transformers
• DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection
• Multitask Learning for Scalable and Dense Multilayer Bayesian Map Inference
• Surround-view Fisheye BEV-Perception for Valet Parking: Dataset, Baseline and Distortion-insensitive Multi-task Framework
• Multitask Learning in Minimally Invasive Surgical Vision: A Review
• MT-CYP-Net: Multi-Task Network for Pixel-Level Crop Yield Prediction Under Very Few Samples
• Multi-task neural networks by learned contextual inputs
• Multi-task Video Enhancement for Dental Interventions
• Estimation of Photometric Redshifts. II. Identification of Out-of-Distribution Data with Neural Networks
• SpeechNet: A Universal Modularized Model for Speech Processing Tasks
• Understanding COVID-19 Vaccine Campaign on Facebook using Minimal Supervision
• Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation
• Multi-task CNN Behavioral Embedding Model For Transaction Fraud Detection
• Weakly Supervised Multi-task Learning for Concept-based Explainability
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• Learning Multiple Dense Prediction Tasks from Partially Annotated Data
• TriangleNet: Edge Prior Augmented Network for Semantic Segmentation through Cross-Task Consistency
• Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task
• Error Detection and Constraint Recovery in Hierarchical Multi-Label Classification without Prior Knowledge
• Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks
• Monocular Spherical Depth Estimation with Explicitly Connected Weak Layout Cues
• FERI: A Multitask-based Fairness Achieving Algorithm with Applications to Fair Organ Transplantation
• SoftNeuro: Fast Deep Inference using Multi-platform Optimization
• Scale-specific auxiliary multi-task contrastive learning for deep liver vessel segmentation
• Multi-path Neural Networks for On-device Multi-domain Visual Classification
• Point, Detect, Count: Multi-Task Medical Image Understanding with Instruction-Tuned Vision-Language Models
• PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context
• Leveraging Road Area Semantic Segmentation with Auxiliary Steering Task
• Towards Robust Partially Supervised Multi-Structure Medical Image Segmentation on Small-Scale Data
• Binaural Audio Generation via Multi-task Learning
• Multi-Task Learning for Robot Perception with Imbalanced Data
• Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNs
• DADO: A Depth-Attention framework for Object Discovery