GAN Cocktail: mixing GANs without dataset access

Abstract

Today's generative models are capable of synthesizing high-fidelity images, but each model specializes on a specific target domain. This raises the need for model merging: combining two or more pretrained generative models into a single unified one. In this work we tackle the problem of model merging, given two constraints that often come up in the real world: (1) no access to the original training data, and (2) without increasing the size of the neural network. To the best of our knowledge, model merging under these constraints has not been studied thus far. We propose a novel, two-stage solution. In the first stage, we transform the weights of all the models to the same parameter space by a technique we term model rooting. In the second stage, we merge the rooted models by averaging their weights and fine-tuning them for each specific domain, using only data generated by the original trained models. We demonstrate that our approach is superior to baseline methods and to existing transfer learning techniques, and investigate several applications.

arXiv commentECCV 2022. Project page is available at: https://omriavrahami.com/GAN-cocktail-page/

References in corpus (17)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Conditional Generative Adversarial Nets
• Overcoming catastrophic forgetting in neural networks
• Learning Transferable Visual Models From Natural Language Supervision
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Zero-Shot Text-to-Image Generation
• Continual Learning Through Synaptic Intelligence
• Mixed Precision Training
• Which Training Methods for GANs do actually Converge?
• Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs
• Continual Learning in Generative Adversarial Nets
• Few-shot Image Generation with Elastic Weight Consolidation