TinyGAN: Distilling BigGAN for Conditional Image Generation

Abstract

Generative Adversarial Networks (GANs) have become a powerful approach for generative image modeling. However, GANs are notorious for their training instability, especially on large-scale, complex datasets. While the recent work of BigGAN has significantly improved the quality of image generation on ImageNet, it requires a huge model, making it hard to deploy on resource-constrained devices. To reduce the model size, we propose a black-box knowledge distillation framework for compressing GANs, which highlights a stable and efficient training process. Given BigGAN as the teacher network, we manage to train a much smaller student network to mimic its functionality, achieving competitive performance on Inception and FID scores with the generator having $16\times$ fewer parameters.

accepted by ACCV 2020

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Compressing Deep Convolutional Networks using Vector Quantization
• Energy-based Generative Adversarial Network
• A Learned Representation For Artistic Style
• Compressing GANs using Knowledge Distillation