Lifelong Generative Modeling

Abstract

Lifelong learning is the problem of learning multiple consecutive tasks in a sequential manner, where knowledge gained from previous tasks is retained and used to aid future learning over the lifetime of the learner. It is essential towards the development of intelligent machines that can adapt to their surroundings. In this work we focus on a lifelong learning approach to unsupervised generative modeling, where we continuously incorporate newly observed distributions into a learned model. We do so through a student-teacher Variational Autoencoder architecture which allows us to learn and preserve all the distributions seen so far, without the need to retain the past data nor the past models. Through the introduction of a novel cross-model regularizer, inspired by a Bayesian update rule, the student model leverages the information learned by the teacher, which acts as a probabilistic knowledge store. The regularizer reduces the effect of catastrophic interference that appears when we learn over sequences of distributions. We validate our model's performance on sequential variants of MNIST, FashionMNIST, PermutedMNIST, SVHN and Celeb-A and demonstrate that our model mitigates the effects of catastrophic interference faced by neural networks in sequential learning scenarios.

32 pages

References in corpus (28)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Overcoming catastrophic forgetting in neural networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Weight Uncertainty in Neural Networks
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Continual Learning Through Synaptic Intelligence
• The Perception-Distortion Tradeoff
• Disentangling by Factorising
• Efficient Lifelong Learning with A-GEM
• Gradient Episodic Memory for Continual Learning
• Progress & Compress: A scalable framework for continual learning
• Variational Continual Learning
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Assessing Generative Models via Precision and Recall
• Deep Generative Dual Memory Network for Continual Learning
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Monte Carlo Gradient Estimation in Machine Learning
• Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies
• Distribution Matching in Variational Inference
• SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
• Continual Classification Learning Using Generative Models
• Associative Compression Networks for Representation Learning
• Learning Attractor Dynamics for Generative Memory

Cited by in corpus (18)

• A continual learning survey: Defying forgetting in classification tasks
• Online Continual Learning with Maximally Interfered Retrieval
• Lifelong Teacher-Student Network Learning
• Continual Classification Learning Using Generative Models
• Continual Learning in Neural Networks
• Learning latent representations across multiple data domains using Lifelong VAEGAN
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain
• Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition, and Selective Transfer
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Transfer Learning via Test-Time Neural Networks Aggregation
• I2CANSAY:Inter-Class Analogical Augmentation and Intra-Class Significance Analysis for Non-Exemplar Online Task-Free Continual Learning
• Kernel-Guided Training of Implicit Generative Models with Stability Guarantees
• Self-Net: Lifelong Learning via Continual Self-Modeling
• Benchmarking Catastrophic Forgetting Mitigation Methods in Federated Time Series Forecasting
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations
• Lifelong Twin Generative Adversarial Networks
• Lifelong Mixture of Variational Autoencoders
• IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay