Which Tasks Should Be Learned Together in Multi-task Learning?

Abstract

Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multi-task learning. This can save computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We study task cooperation and competition in several different learning settings and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.

Presented to ICML 2020 See project website at http://taskgrouping.stanford.edu/

Cited by in corpus (41)

• Multi-Task Learning for Dense Prediction Tasks: A Survey
• Models Genesis
• Multi-task head pose estimation in-the-wild
• Multi-task learning for natural language processing in the 2020s: where are we going?
• Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning
• Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings
• Replay in Deep Learning: Current Approaches and Missing Biological Elements
• Controllable Pareto Multi-Task Learning
• Multi-Modal Multi-Task (3MT) Road Segmentation
• Multitask Learning for Scalable and Dense Multilayer Bayesian Map Inference
• Deep multi-task mining Calabi-Yau four-folds
• A Meta-Learning Approach for Graph Representation Learning in Multi-Task Settings
• Multi-task learning for virtual flow metering
• WTR: A Test Collection for Web Table Retrieval
• Improving Vessel Segmentation with Multi-Task Learning and Auxiliary Data Available Only During Model Training
• On the Complementarity of Data Selection and Fine Tuning for Domain Adaptation
• Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis
• GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning
• PVG at WASSA 2021: A Multi-Input, Multi-Task, Transformer-Based Architecture for Empathy and Distress Prediction
• STG-MTL: Scalable Task Grouping for Multi-Task Learning Using Data Map
• Multi-Task Variational Information Bottleneck
• Boosting Multitask Learning on Graphs through Higher-Order Task Affinities
• Evaluating the structure of cognitive tasks with transfer learning
• TriangleNet: Edge Prior Augmented Network for Semantic Segmentation through Cross-Task Consistency
• Regularising disparity estimation via multi task learning with structured light reconstruction
• Fisher Task Distance and Its Application in Neural Architecture Search
• RotoGrad: Gradient Homogenization in Multitask Learning
• DEF: Deep Estimation of Sharp Geometric Features in 3D Shapes
• Frosting Weights for Better Continual Training
• Taxonomy Completion via Triplet Matching Network
• Abusive Language Detection in Heterogeneous Contexts: Dataset Collection and the Role of Supervised Attention
• Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task
• Task Affinity with Maximum Bipartite Matching in Few-Shot Learning
• Learning Interpretable Microscopic Features of Tumor by Multi-task Adversarial CNNs To Improve Generalization
• Compressed Hierarchical Representations for Multi-Task Learning and Task Clustering
• Generative Modeling for Multi-task Visual Learning
• Unlocking the Full Potential of Small Data with Diverse Supervision
• Unified Representation Learning for Efficient Medical Image Analysis
• Disentangling Transfer and Interference in Multi-Domain Learning