Teachers Do More Than Teach: Compressing Image-to-Image Models

Abstract

Generative Adversarial Networks (GANs) have achieved huge success in generating high-fidelity images, however, they suffer from low efficiency due to tremendous computational cost and bulky memory usage. Recent efforts on compression GANs show noticeable progress in obtaining smaller generators by sacrificing image quality or involving a time-consuming searching process. In this work, we aim to address these issues by introducing a teacher network that provides a search space in which efficient network architectures can be found, in addition to performing knowledge distillation. First, we revisit the search space of generative models, introducing an inception-based residual block into generators. Second, to achieve target computation cost, we propose a one-step pruning algorithm that searches a student architecture from the teacher model and substantially reduces searching cost. It requires no l1 sparsity regularization and its associated hyper-parameters, simplifying the training procedure. Finally, we propose to distill knowledge through maximizing feature similarity between teacher and student via an index named Global Kernel Alignment (GKA). Our compressed networks achieve similar or even better image fidelity (FID, mIoU) than the original models with much-reduced computational cost, e.g., MACs. Code will be released at https://github.com/snap-research/CAT.

arXiv comment18 pages, 10 figures, accepted by CVPR 2021

References in corpus (28)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Neural Architecture Search with Reinforcement Learning
• Knowledge Distillation: A Survey
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Self-Attention Generative Adversarial Networks
• DARTS: Differentiable Architecture Search
• Rethinking the Value of Network Pruning
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Large-Scale Evolution of Image Classifiers
• Similarity of Neural Network Representations Revisited
• Algorithms for Learning Kernels Based on Centered Alignment
• Neural Architecture Transfer
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis
• AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks
• Compressing GANs using Knowledge Distillation
• Towards Efficient Training for Neural Network Quantization
• Non-Structured DNN Weight Pruning -- Is It Beneficial in Any Platform?
• 26ms Inference Time for ResNet-50: Towards Real-Time Execution of all DNNs on Smartphone
• Rethinking Spatially-Adaptive Normalization
• Toward Extremely Low Bit and Lossless Accuracy in DNNs with Progressive ADMM
• CoCoPIE: Making Mobile AI Sweet As PIE --Compression-Compilation Co-Design Goes a Long Way
• Human Motion Transfer from Poses in the Wild
• FracBits: Mixed Precision Quantization via Fractional Bit-Widths