Compressing GANs using Knowledge Distillation

Abstract

Generative Adversarial Networks (GANs) have been used in several machine learning tasks such as domain transfer, super resolution, and synthetic data generation. State-of-the-art GANs often use tens of millions of parameters, making them expensive to deploy for applications in low SWAP (size, weight, and power) hardware, such as mobile devices, and for applications with real time capabilities. There has been no work found to reduce the number of parameters used in GANs. Therefore, we propose a method to compress GANs using knowledge distillation techniques, in which a smaller "student" GAN learns to mimic a larger "teacher" GAN. We show that the distillation methods used on MNIST, CIFAR-10, and Celeb-A datasets can compress teacher GANs at ratios of 1669:1, 58:1, and 87:1, respectively, while retaining the quality of the generated image. From our experiments, we observe a qualitative limit for GAN's compression. Moreover, we observe that, with a fixed parameter budget, compressed GANs outperform GANs trained using standard training methods. We conjecture that this is partially owing to the optimization landscape of over-parameterized GANs which allows efficient training using alternating gradient descent. Thus, training an over-parameterized GAN followed by our proposed compression scheme provides a high quality generative model with a small number of parameters.

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?

Cited by in corpus (15)

• Knowledge Distillation: A Survey
• StyleGAN2 Distillation for Feed-forward Image Manipulation
• PyNET-QxQ: An Efficient PyNET Variant for QxQ Bayer Pattern Demosaicing in CMOS Image Sensors
• Anycost GANs for Interactive Image Synthesis and Editing
• Teachers Do More Than Teach: Compressing Image-to-Image Models
• Self-Supervised GAN Compression
• Distilling portable Generative Adversarial Networks for Image Translation
• TinyGAN: Distilling BigGAN for Conditional Image Generation
• Content-Aware GAN Compression
• Slimmable Generative Adversarial Networks
• Evolutionary Generative Adversarial Networks with Crossover Based Knowledge Distillation
• A Survey on GAN Acceleration Using Memory Compression Technique
• Class-dependent Compression of Deep Neural Networks
• Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation
• Transferring Knowledge with Attention Distillation for Multi-Domain Image-to-Image Translation