CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks

Abstract

We present a continual learning approach for generative adversarial networks (GANs), by designing and leveraging parameter-efficient feature map transformations. Our approach is based on learning a set of global and task-specific parameters. The global parameters are fixed across tasks whereas the task-specific parameters act as local adapters for each task, and help in efficiently obtaining task-specific feature maps. Moreover, we propose an element-wise addition of residual bias in the transformed feature space, which further helps stabilize GAN training in such settings. Our approach also leverages task similarity information based on the Fisher information matrix. Leveraging this knowledge from previous tasks significantly improves the model performance. In addition, the similarity measure also helps reduce the parameter growth in continual adaptation and helps to learn a compact model. In contrast to the recent approaches for continually-learned GANs, the proposed approach provides a memory-efficient way to perform effective continual data generation. Through extensive experiments on challenging and diverse datasets, we show that the feature-map-transformation approach outperforms state-of-the-art methods for continually-learned GANs, with substantially fewer parameters. The proposed method generates high-quality samples that can also improve the generative-replay-based continual learning for discriminative tasks.

arXiv commentUnder Submission

References in corpus (12)

• Auto-Encoding Variational Bayes
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Overcoming catastrophic forgetting in neural networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• How transferable are features in deep neural networks?
• Learning Transferable Features with Deep Adaptation Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Which Training Methods for GANs do actually Converge?
• Three scenarios for continual learning
• Reinforced Continual Learning
• Generative replay with feedback connections as a general strategy for continual learning
• Less-forgetting Learning in Deep Neural Networks