AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

Abstract

Multi-task learning is an open and challenging problem in computer vision. The typical way of conducting multi-task learning with deep neural networks is either through handcrafted schemes that share all initial layers and branch out at an adhoc point, or through separate task-specific networks with an additional feature sharing/fusion mechanism. Unlike existing methods, we propose an adaptive sharing approach, called AdaShare, that decides what to share across which tasks to achieve the best recognition accuracy, while taking resource efficiency into account. Specifically, our main idea is to learn the sharing pattern through a task-specific policy that selectively chooses which layers to execute for a given task in the multi-task network. We efficiently optimize the task-specific policy jointly with the network weights, using standard back-propagation. Experiments on several challenging and diverse benchmark datasets with a variable number of tasks well demonstrate the efficacy of our approach over state-of-the-art methods. Project page: https://cs-people.bu.edu/sunxm/AdaShare/project.html.

arXiv commentNeurips 2020 camera-ready version

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• An Overview of Multi-Task Learning in Deep Neural Networks
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• Theoretical Models of Learning to Learn
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Large-Scale Evolution of Image Classifiers
• Hierarchical Multi-Scale Attention for Semantic Segmentation
• Clustered Multi-Task Learning: A Convex Formulation
• Deep Networks with Stochastic Depth
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Conditional Computation in Neural Networks for faster models
• Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
• LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition
• Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering
• Exploring Shared Structures and Hierarchies for Multiple NLP Tasks
• Real-time Semantic Segmentation with Fast Attention

Cited by in corpus (19)

• Multi-Task Learning with Deep Neural Networks: A Survey
• A Multi-modal Fusion Framework Based on Multi-task Correlation Learning for Cancer Prognosis Prediction
• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
• Learning to Branch for Multi-Task Learning
• Parameter-Efficient Transfer Learning with Diff Pruning
• DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning
• Measuring and Harnessing Transference in Multi-Task Learning
• STG-MTL: Scalable Task Grouping for Multi-Task Learning Using Data Map
• AdapMTL: Adaptive Pruning Framework for Multitask Learning Model
• Adaptive Weight Assignment Scheme For Multi-task Learning
• Latent Domain Learning with Dynamic Residual Adapters
• Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images
• T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler
• RotoGrad: Gradient Homogenization in Multitask Learning
• Multi-path Neural Networks for On-device Multi-domain Visual Classification
• Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
• Generative Modeling for Multi-task Visual Learning
• Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
• Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints