Lifelong Language Knowledge Distillation
arXiv:2010.02123
Abstract
It is challenging to perform lifelong language learning (LLL) on a stream of different tasks without any performance degradation comparing to the multi-task counterparts. To address this issue, we present Lifelong Language Knowledge Distillation (L2KD), a simple but efficient method that can be easily applied to existing LLL architectures in order to mitigate the degradation. Specifically, when the LLL model is trained on a new task, we assign a teacher model to first learn the new task, and pass the knowledge to the LLL model via knowledge distillation. Therefore, the LLL model can better adapt to the new task while keeping the previously learned knowledge. Experiments show that the proposed L2KD consistently improves previous state-of-the-art models, and the degradation comparing to multi-task models in LLL tasks is well mitigated for both sequence generation and text classification tasks.
arXiv commentEMNLP 2020 long paper
References in corpus (10)
- Distilling the Knowledge in a Neural Network
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
- Non-Autoregressive Neural Machine Translation
- Gradient Episodic Memory for Continual Learning
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Generative replay with feedback connections as a general strategy for continual learning
- Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
- Toward Continual Learning for Conversational Agents
- Lifelong Domain Word Embedding via Meta-Learning
- Continual Learning for Sentence Representations Using Conceptors