Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering
arXiv:1711.00108
Abstract
Existing deep multitask learning (MTL) approaches align layers shared between tasks in a parallel ordering. Such an organization significantly constricts the types of shared structure that can be learned. The necessity of parallel ordering for deep MTL is first tested by comparing it with permuted ordering of shared layers. The results indicate that a flexible ordering can enable more effective sharing, thus motivating the development of a soft ordering approach, which learns how shared layers are applied in different ways for different tasks. Deep MTL with soft ordering outperforms parallel ordering methods across a series of domains. These results suggest that the power of deep MTL comes from learning highly general building blocks that can be assembled to meet the demands of each task.
arXiv comment14 pages (main paper: 10 pages). Published as a conference paper at ICLR 2018
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- PathNet: Evolution Channels Gradient Descent in Super Neural Networks
- Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex
- Universal representations:The missing link between faces, text, planktons, and cat breeds
- Deep Multi-task Representation Learning: A Tensor Factorisation Approach
- Deep Learning Face Attributes in the Wild
- A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
- Improved Dropout for Shallow and Deep Learning
Cited by in corpus (8)
- Multi-Task Learning with Deep Neural Networks: A Survey
- AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
- Recent Advances of Continual Learning in Computer Vision: An Overview
- Modular meta-learning
- Exploring Shared Structures and Hierarchies for Multiple NLP Tasks
- Feature Partitioning for Efficient Multi-Task Architectures
- Modular meta-learning in abstract graph networks for combinatorial generalization
- Fast Line Search for Multi-Task Learning