Large-Scale Generative Data-Free Distillation
arXiv:2012.05578
Abstract
Knowledge distillation is one of the most popular and effective techniques for knowledge transfer, model compression and semi-supervised learning. Most existing distillation approaches require the access to original or augmented training samples. But this can be problematic in practice due to privacy, proprietary and availability concerns. Recent work has put forward some methods to tackle this problem, but they are either highly time-consuming or unable to scale to large datasets. To this end, we propose a new method to train a generative image model by leveraging the intrinsic normalization layers' statistics of the trained teacher network. This enables us to build an ensemble of generators without training data that can efficiently produce substitute inputs for subsequent distillation. The proposed method pushes forward the data-free distillation performance on CIFAR-10 and CIFAR-100 to 95.02% and 77.02% respectively. Furthermore, we are able to scale it to ImageNet dataset, which to the best of our knowledge, has never been done using generative models in a data-free setting.
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- mixup: Beyond Empirical Risk Minimization
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Federated Learning: Strategies for Improving Communication Efficiency
- FitNets: Hints for Thin Deep Nets
- GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- AutoAugment: Learning Augmentation Policies from Data
- Data-Free Knowledge Distillation for Deep Neural Networks
- Adaptive Gradient Methods with Dynamic Bound of Learning Rate
- Data-Free Adversarial Distillation
- Moonshine: Distilling with Cheap Convolutions
- Zero-Shot Knowledge Distillation in Deep Networks
- Self-Knowledge Distillation in Natural Language Processing