Continual Learning of Generative Models with Limited Data: From Wasserstein-1 Barycenter to Adaptive Coalescence

Abstract

Learning generative models is challenging for a network edge node with limited data and computing power. Since tasks in similar environments share model similarity, it is plausible to leverage pre-trained generative models from the cloud or other edge nodes. Appealing to optimal transport theory tailored towards Wasserstein-1 generative adversarial networks (WGAN), this study aims to develop a framework which systematically optimizes continual learning of generative models using local data at the edge node while exploiting adaptive coalescence of pre-trained generative models. Specifically, by treating the knowledge transfer from other nodes as Wasserstein balls centered around their pre-trained models, continual learning of generative models is cast as a constrained optimization problem, which is further reduced to a Wasserstein-1 barycenter problem. A two-stage approach is devised accordingly: 1) The barycenters among the pre-trained models are computed offline, where displacement interpolation is used as the theoretic foundation for finding adaptive barycenters via a "recursive" WGAN configuration; 2) the barycenter computed offline is used as meta-model initialization for continual learning and then fast adaptation is carried out to find the generative model using the local samples at the target edge node. Finally, a weight ternarization method, based on joint optimization of weights and threshold for quantization, is developed to compress the generative model further.

References in corpus (17)

• Overcoming catastrophic forgetting in neural networks
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• A continual learning survey: Defying forgetting in classification tasks
• Continual Learning Through Synaptic Intelligence
• Fast Computation of Wasserstein Barycenters
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• Stabilizing GAN Training with Multiple Random Projections
• Decentralized Learning of Generative Adversarial Networks from Non-iid Data
• SquishedNets: Squishing SqueezeNet further for edge device scenarios via deep evolutionary synthesis
• A domain agnostic measure for monitoring and evaluating GANs
• A Geometric View of Optimal Transportation and Generative Model
• Real-Time Edge Intelligence in the Making: A Collaborative Learning Framework via Federated Meta-Learning
• Adversarial Computation of Optimal Transport Maps
• Model Fusion via Optimal Transport