LAMOL: LAnguage MOdeling for Lifelong Language Learning

Abstract

Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudo-samples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at https://github.com/jojotenya/LAMOL.

References in corpus (16)

• Overcoming catastrophic forgetting in neural networks
• Character-level Convolutional Networks for Text Classification
• Continual Learning Through Synaptic Intelligence
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• Learning to Generate Reviews and Discovering Sentiment
• The Natural Language Decathlon: Multitask Learning as Question Answering
• FearNet: Brain-Inspired Model for Incremental Learning
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Continual Learning with Deep Generative Replay
• A Network-based End-to-End Trainable Task-oriented Dialogue System
• Net2Net: Accelerating Learning via Knowledge Transfer
• Toward Continual Learning for Conversational Agents
• Towards Training Recurrent Neural Networks for Lifelong Learning

Cited by in corpus (15)

• Continual Learning for Recurrent Neural Networks: an Empirical Evaluation
• Towards Continual Knowledge Learning of Language Models
• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5
• Mind the Gap: Assessing Temporal Generalization in Neural Language Models
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• DARE: Data Augmented Relation Extraction with GPT-2
• One Person, One Model, One World: Learning Continual User Representation without Forgetting
• Meta-Learning with Sparse Experience Replay for Lifelong Language Learning
• DRILL: Dynamic Representations for Imbalanced Lifelong Learning
• Update Frequently, Update Fast: Retraining Semantic Parsing Systems in a Fraction of Time
• Continual Learning for Text Classification with Information Disentanglement Based Regularization
• Curriculum-Meta Learning for Order-Robust Continual Relation Extraction
• Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning
• Online Continual Learning in Image Classification: An Empirical Survey
• Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates