Feature Partitioning for Efficient Multi-Task Architectures
arXiv:1908.04339
Abstract
Multi-task learning holds the promise of less data, parameters, and time than training of separate models. We propose a method to automatically search over multi-task architectures while taking resource constraints into consideration. We propose a search space that compactly represents different parameter sharing strategies. This provides more effective coverage and sampling of the space of multi-task architectures. We also present a method for quick evaluation of different architectures by using feature distillation. Together these contributions allow us to quickly optimize for efficient multi-task models. We benchmark on Visual Decathlon, demonstrating that we can automatically search for and identify multi-task architectures that effectively make trade-offs between task resource requirements while achieving a high level of final performance.
References in corpus (24)
- Neural Architecture Search with Reinforcement Learning
- An Overview of Multi-Task Learning in Deep Neural Networks
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Efficient Neural Architecture Search via Parameter Sharing
- A Survey on Multi-Task Learning
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Learning multiple visual domains with residual adapters
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- One Model To Learn Them All
- Random Search and Reproducibility for Neural Architecture Search
- Hierarchical Representations for Efficient Architecture Search
- Progressive Neural Architecture Search
- Simple random search provides a competitive approach to reinforcement learning
- Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
- Peephole: Predicting Network Performance Before Training
- Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
- Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
- Multi-Task Zipping via Layer-wise Neuron Sharing
- End-to-End Multi-Task Learning with Attention
- Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering
- Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
- Learning to Multi-Task by Active Sampling
- Evolutionary Architecture Search For Deep Multitask Networks
- Unifying and Merging Well-trained Deep Neural Networks for Inference Stage
Cited by in corpus (7)
- Multi-Task Learning for Dense Prediction Tasks: A Survey
- Multi-Task Learning with Deep Neural Networks: A Survey
- Text as Neural Operator: Image Manipulation by Text Instruction
- Pose Neural Fabrics Search
- Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks
- Multi-path Neural Networks for On-device Multi-domain Visual Classification
- What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning