Continual Learning for Recurrent Neural Networks: an Empirical Evaluation

Abstract

Learning continuously during all model lifetime is fundamental to deploy machine learning solutions robust to drifts in the data distribution. Advances in Continual Learning (CL) with recurrent neural networks could pave the way to a large number of applications where incoming data is non stationary, like natural language processing and robotics. However, the existing body of work on the topic is still fragmented, with approaches which are application-specific and whose assessment is based on heterogeneous learning protocols and datasets. In this paper, we organize the literature on CL for sequential data processing by providing a categorization of the contributions and a review of the benchmarks. We propose two new benchmarks for CL with sequential data based on existing datasets, whose characteristics resemble real-world applications. We also provide a broad empirical evaluation of CL and Recurrent Neural Networks in class-incremental scenario, by testing their ability to mitigate forgetting with a number of different strategies which are not specific to sequential data processing. Our results highlight the key role played by the sequence length and the importance of a clear specification of the CL scenario.

arXiv commentPublished in Neural Networks

References in corpus (51)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Neural Machine Translation by Jointly Learning to Align and Translate
• Distilling the Knowledge in a Neural Network
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Overcoming catastrophic forgetting in neural networks
• A continual learning survey: Defying forgetting in classification tasks
• Sequence Transduction with Recurrent Neural Networks
• Recent Trends in Deep Learning Based Natural Language Processing
• Progressive Neural Networks
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• Efficient Lifelong Learning with A-GEM
• Three scenarios for continual learning
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• A Neural Representation of Sketch Drawings
• Experience Replay for Continual Learning
• Progress & Compress: A scalable framework for continual learning
• Variational Continual Learning
• Continuous online sequence learning with an unsupervised neural network model
• Generative replay with feedback connections as a general strategy for continual learning
• Towards Robust Evaluations of Continual Learning
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Uncertainty-based Continual Learning with Adaptive Regularization
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• An Empirical Study of Example Forgetting during Deep Neural Network Learning
• Online Meta-Learning
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organization
• Continual Learning via Neural Pruning
• Selfless Sequential Learning
• Task Agnostic Continual Learning Using Online Variational Bayes
• Learning to Continually Learn
• SpaceNet: Make Free Space For Continual Learning
• Task Agnostic Continual Learning via Meta Learning
• How do neurons operate on sparse distributed representations? A mathematical theory of sparsity, neurons and active dendrites
• Interpreting recurrent neural networks behaviour via excitable network attractors
• A Unifying Bayesian View of Continual Learning
• Continual Learning with Gated Incremental Memories for sequential data processing
• Regularization Shortcomings for Continual Learning
• Progressive Memory Banks for Incremental Domain Adaptation
• Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning
• Continual Learning in Recurrent Neural Networks
• Incremental Sequence Learning
• Continuous Meta-Learning without Tasks
• A Multi-Task Learning Framework for Overcoming the Catastrophic Forgetting in Automatic Speech Recognition
• Evaluating Online Continual Learning with CALM
• Spiking Neural Predictive Coding for Continual Learning from Data Streams
• Continual Learning Using Bayesian Neural Networks
• Making Good on LSTMs' Unfulfilled Promise
• Continual Learning of New Sound Classes using Generative Replay
• Catastrophic Forgetting in Deep Graph Networks: an Introductory Benchmark for Graph Classification

Cited by in corpus (8)

• Continual Learning for Predictive Maintenance: Overview and Challenges
• Lifelong Continual Learning for Anomaly Detection: New Challenges, Perspectives, and Insights
• Continual learning of longitudinal health records
• Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition
• Continual Human-in-the-Loop Optimization
• Continual Policy Distillation of Reinforcement Learning-based Controllers for Soft Robotic In-Hand Manipulation
• Towards lifelong learning of Recurrent Neural Networks for control design
• Continual Learning of Multi-modal Dynamics with External Memory