MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning
arXiv:2003.14058
Abstract
We propose to incorporate neural architecture search (NAS) into general-purpose multi-task learning (GP-MTL). Existing NAS methods typically define different search spaces according to different tasks. In order to adapt to different task combinations (i.e., task sets), we disentangle the GP-MTL networks into single-task backbones (optionally encode the task priors), and a hierarchical and layerwise features sharing/fusing scheme across them. This enables us to design a novel and general task-agnostic search space, which inserts cross-task edges (i.e., feature fusion connections) into fixed single-task network backbones. Moreover, we also propose a novel single-shot gradient-based search algorithm that closes the performance gap between the searched architectures and the final evaluation architecture. This is realized with a minimum entropy regularization on the architecture weights during the search phase, which makes the architecture weights converge to near-discrete values and therefore achieves a single model. As a result, our searched model can be directly used for evaluation without (re-)training from scratch. We perform extensive experiments using different single-task backbones on various task sets, demonstrating the promising performance obtained by exploiting the hierarchical and layerwise features, as well as the desirable generalizability to different i) task sets and ii) single-task backbones. The code of our paper is available at https://github.com/bhpfelix/MTLNAS.
arXiv commentAccepted to CVPR2020. The first two authors contribute equally
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Neural Architecture Search with Reinforcement Learning
- An Overview of Multi-Task Learning in Deep Neural Networks
- Neural Architecture Search: A Survey
- DARTS: Differentiable Architecture Search
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Efficient Neural Architecture Search via Parameter Sharing
- Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
- Beyond Skip Connections: Top-Down Modulation for Object Detection
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- One-Shot Neural Architecture Search via Self-Evaluated Template Network
- Simple And Efficient Architecture Search for Convolutional Neural Networks
- DetNAS: Backbone Search for Object Detection
- Convolutional Neural Fabrics
- Deep Multi-task Representation Learning: A Tensor Factorisation Approach
- Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search
- Multi-Task Learning by Deep Collaboration and Application in Facial Landmark Detection