Meta-Learning Multi-task Communication
arXiv:1810.09988
Abstract
In this paper, we describe a general framework: Parameters Read-Write Networks (PRaWNs) to systematically analyze current neural models for multi-task learning, in which we find that existing models expect to disentangle features into different spaces while features learned in practice are still entangled in shared space, leaving potential hazards for other training or unseen tasks. We propose to alleviate this problem by incorporating an inductive bias into the process of multi-task learning, that each task can keep informed of not only the knowledge stored in other tasks but the way how other tasks maintain their knowledge. In practice, we achieve above inductive bias by allowing different tasks to communicate by passing both hidden variables and gradients explicitly. Experimentally, we evaluate proposed methods on three groups of tasks and two types of settings (\textsc{in-task} and \textsc{out-of-task}). Quantitative and qualitative results show their effectiveness.
arXiv commentA related blog can be found on the author's homepage
References in corpus (9)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- ADADELTA: An Adaptive Learning Rate Method
- Recurrent Neural Network for Text Classification with Multi-Task Learning
- One-shot Learning with Memory-Augmented Neural Networks
- Multi-Task Cross-Lingual Sequence Tagging from Scratch
- One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
- Deep Multi-task Representation Learning: A Tensor Factorisation Approach
- Meta Multi-Task Learning for Sequence Modeling
- Exploring Shared Structures and Hierarchies for Multiple NLP Tasks