Efficient Continual Learning in Neural Networks with Embedding Regularization

Abstract

Continual learning of deep neural networks is a key requirement for scaling them up to more complex applicative scenarios and for achieving real lifelong learning of these architectures. Previous approaches to the problem have considered either the progressive increase in the size of the networks, or have tried to regularize the network behavior to equalize it with respect to previously observed tasks. In the latter case, it is essential to understand what type of information best represents this past behavior. Common techniques include regularizing the past outputs, gradients, or individual weights. In this work, we propose a new, relatively simple and efficient method to perform continual learning by regularizing instead the network internal embeddings. To make the approach scalable, we also propose a dynamic sampling strategy to reduce the memory footprint of the required external storage. We show that our method performs favorably with respect to state-of-the-art approaches in the literature, while requiring significantly less space in memory and computational time. In addition, inspired inspired by to recent works, we evaluate the impact of selecting a more flexible model for the activation functions inside the network, evaluating the impact of catastrophic forgetting on the activation functions themselves.

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Continual Learning Through Synaptic Intelligence
• Deep Image Prior
• Efficient Lifelong Learning with A-GEM
• Learning Activation Functions to Improve Deep Neural Networks
• Progress & Compress: A scalable framework for continual learning
• On Quadratic Penalties in Elastic Weight Consolidation
• Don't forget, there is more than forgetting: new metrics for Continual Learning
• Deep Learning with S-shaped Rectified Linear Activation Units
• Pseudo-Recursal: Solving the Catastrophic Forgetting Problem in Deep Neural Networks
• Neural Graph Machines: Learning Neural Networks Using Graphs
• Universal Intelligence: A Definition of Machine Intelligence

Cited by in corpus (7)

• SpaceNet: Make Free Space For Continual Learning
• Practical Recommendations for Replay-based Continual Learning Methods
• The Effectiveness of Memory Replay in Large Scale Continual Learning
• Pseudo-Rehearsal for Continual Learning with Normalizing Flows
• Task-agnostic Continual Learning with Hybrid Probabilistic Models
• GROWN: GRow Only When Necessary for Continual Learning
• Unsupervised Continual Learning Via Pseudo Labels