Towards Training Recurrent Neural Networks for Lifelong Learning

Abstract

Catastrophic forgetting and capacity saturation are the central challenges of any parametric lifelong learning system. In this work, we study these challenges in the context of sequential supervised learning with an emphasis on recurrent neural networks. To evaluate the models in the lifelong learning setting, we propose a curriculum-based, simple, and intuitive benchmark where the models are trained on tasks with increasing levels of difficulty. To measure the impact of catastrophic forgetting, the model is tested on all the previous tasks as it completes any task. As a step towards developing true lifelong learning systems, we unify Gradient Episodic Memory (a catastrophic forgetting alleviation approach) and Net2Net(a capacity expansion approach). Both these models are proposed in the context of feedforward networks and we evaluate the feasibility of using them for recurrent networks. Evaluation on the proposed benchmark shows that the unified model is more suitable than the constituent models for lifelong learning setting.

arXiv commentAccepted at Neural Computation

References in corpus (5)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• FitNets: Hints for Thin Deep Nets
• Net2Net: Accelerating Learning via Knowledge Transfer
• Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting

Cited by in corpus (8)

• Lifelong Sequential Modeling with Personalized Memorization for User Response Prediction
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• Continual Learning with Gated Incremental Memories for sequential data processing
• Neural Architecture Search for Class-incremental Learning
• Dynamic Memory based Attention Network for Sequential Recommendation
• Continual Representation Learning for Biometric Identification
• Layerwise Optimization by Gradient Decomposition for Continual Learning
• TAG: Task-based Accumulated Gradients for Lifelong learning