Memory Replay GANs: learning to generate images from new categories without forgetting

Abstract

Previous works on sequential learning address the problem of forgetting in discriminative models. In this paper we consider the case of generative models. In particular, we investigate generative adversarial networks (GANs) in the task of learning new categories in a sequential fashion. We first show that sequential fine tuning renders the network unable to properly generate images from previous categories (i.e. forgetting). Addressing this problem, we propose Memory Replay GANs (MeRGANs), a conditional GAN framework that integrates a memory replay generator. We study two methods to prevent forgetting by leveraging these replays, namely joint training with replay and replay alignment. Qualitative and quantitative experimental results in MNIST, SVHN and LSUN datasets show that our memory replay approach can generate competitive images while significantly mitigating the forgetting of previous categories.

arXiv commentAppear in NeurIPS 2018

References in corpus (12)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Conditional Generative Adversarial Nets
• Overcoming catastrophic forgetting in neural networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Conditional Image Synthesis With Auxiliary Classifier GANs
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• A Learned Representation For Artistic Style
• cGANs with Projection Discriminator
• FearNet: Brain-Inspired Model for Incremental Learning
• Continual Learning in Generative Adversarial Nets
• Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting
• Transferring GANs: generating images from limited data

Cited by in corpus (24)

• Variational Federated Multi-Task Learning
• Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
• DisCoRL: Continual Reinforcement Learning via Policy Distillation
• Multi-Task Incremental Learning for Object Detection
• Replay in Deep Learning: Current Approaches and Missing Biological Elements
• Continual State Representation Learning for Reinforcement Learning using Generative Replay
• Rainbow Memory: Continual Learning with a Memory of Diverse Samples
• Progressive Memory Banks for Incremental Domain Adaptation
• Semantic Drift Compensation for Class-Incremental Learning
• Carousel Memory: Rethinking the Design of Episodic Memory for Continual Learning
• Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
• Generative Feature Replay For Class-Incremental Learning
• Self-Supervised Learning Aided Class-Incremental Lifelong Learning
• On the importance of cross-task features for class-incremental learning
• Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference
• PKI: Prior Knowledge-Infused Neural Network for Few-Shot Class-Incremental Learning
• Lifelong Vehicle Trajectory Prediction Framework Based on Generative Replay
• Incremental Concept Learning via Online Generative Memory Recall
• Closed-Loop Memory GAN for Continual Learning
• ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning
• Self-Improving Semantic Perception for Indoor Localisation
• Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation
• Lifelong Learning Process: Self-Memory Supervising and Dynamically Growing Networks
• Shared and Private VAEs with Generative Replay for Continual Learning