Learning to Branch for Multi-Task Learning

Abstract

Training multiple tasks jointly in one deep network yields reduced latency during inference and better performance over the single-task counterpart by sharing certain layers of a network. However, over-sharing a network could erroneously enforce over-generalization, causing negative knowledge transfer across tasks. Prior works rely on human intuition or pre-computed task relatedness scores for ad hoc branching structures. They provide sub-optimal end results and often require huge efforts for the trial-and-error process. In this work, we present an automated multi-task learning algorithm that learns where to share or branch within a network, designing an effective network topology that is directly optimized for multiple objectives across tasks. Specifically, we propose a novel tree-structured design space that casts a tree branching operation as a gumbel-softmax sampling procedure. This enables differentiable network splitting that is end-to-end trainable. We validate the proposed method on controlled synthetic data, CelebA, and Taskonomy.

Accepted at ICML 2020

References in corpus (11)

• Categorical Reparameterization with Gumbel-Softmax
• An Overview of Multi-Task Learning in Deep Neural Networks
• DARTS: Differentiable Architecture Search
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Efficient Neural Architecture Search via Parameter Sharing
• Multi-Task Learning as Multi-Objective Optimization
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• Transfer Learning with Neural AutoML
• Meta Architecture Search
• Walk and Learn: Facial Attribute Representation Learning from Egocentric Video and Contextual Data
• Flexible Multi-task Networks by Learning Parameter Allocation

Cited by in corpus (8)

• Efficiently Identifying Task Groupings for Multi-Task Learning
• Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models
• Measuring and Harnessing Transference in Multi-Task Learning
• Empirical Study of Multi-Task Hourglass Model for Semantic Segmentation Task
• Scalable Multitask Learning Using Gradient-based Estimation of Task Affinity
• High-Capacity Expert Binary Networks
• Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks
• Multi-path Neural Networks for On-device Multi-domain Visual Classification