GAN Memory with No Forgetting
arXiv:2006.07543
Abstract
As a fundamental issue in lifelong learning, catastrophic forgetting is directly caused by inaccessible historical data; accordingly, if the data (information) were memorized perfectly, no forgetting should be expected. Motivated by that, we propose a GAN memory for lifelong learning, which is capable of remembering a stream of datasets via generative processes, with \emph{no} forgetting. Our GAN memory is based on recognizing that one can modulate the "style" of a GAN model to form perceptually-distant targeted generation. Accordingly, we propose to do sequential style modulations atop a well-behaved base GAN model, to form sequential targeted generative models, while simultaneously benefiting from the transferred base knowledge. The GAN memory -- that is motivated by lifelong learning -- is therefore itself manifested by a form of lifelong learning, via forward transfer and modulation of information from prior tasks. Experiments demonstrate the superiority of our method over existing approaches and its effectiveness in alleviating catastrophic forgetting for lifelong classification problems. Code is available at https://github.com/MiaoyunZhao/GANmemory_LifelongLearning.
arXiv commentNeurIPS2020
References in corpus (35)
- Conditional Generative Adversarial Nets
- Overcoming catastrophic forgetting in neural networks
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- How transferable are features in deep neural networks?
- Spectral Normalization for Generative Adversarial Networks
- Learning Transferable Features with Deep Adaptation Networks
- The Effectiveness of Data Augmentation in Image Classification using Deep Learning
- GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- Continual Learning Through Synaptic Intelligence
- Which Training Methods for GANs do actually Converge?
- A Learned Representation For Artistic Style
- MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
- Three scenarios for continual learning
- Data Synthesis based on Generative Adversarial Networks
- GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
- MaskGAN: Better Text Generation via Filling in the______
- Variational Continual Learning
- Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization
- Generative replay with feedback connections as a general strategy for continual learning
- GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
- K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
- Video-to-Video Synthesis
- Continual Learning in Generative Adversarial Nets
- On Leveraging Pretrained GANs for Generation with Limited Data
- Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition
- Transfer Fine-Tuning: A BERT Case Study
- ZM-Net: Real-time Zero-shot Image Manipulation Network
- Continual State Representation Learning for Reinforcement Learning using Generative Replay
- HUBERT Untangles BERT to Improve Transfer across NLP Tasks
- Probability density distillation with generative adversarial networks for high-quality parallel waveform generation
- Open Event Extraction from Online Text using a Generative Adversarial Network