Data-Free Adversarial Distillation

Abstract

Knowledge Distillation (KD) has made remarkable progress in the last few years and become a popular paradigm for model compression and knowledge transfer. However, almost all existing KD algorithms are data-driven, i.e., relying on a large amount of original training data or alternative data, which is usually unavailable in real-world scenarios. In this paper, we devote ourselves to this challenging problem and propose a novel adversarial distillation mechanism to craft a compact student model without any real-world data. We introduce a model discrepancy to quantificationally measure the difference between student and teacher models and construct an optimizable upper bound. In our work, the student and the teacher jointly act the role of the discriminator to reduce this discrepancy, when a generator adversarially produces some "hard samples" to enlarge it. Extensive experiments demonstrate that the proposed data-free method yields comparable performance to existing data-driven methods. More strikingly, our approach can be directly extended to semantic segmentation, which is more complicated than classification, and our approach achieves state-of-the-art results. Code and pretrained models are available at https://github.com/VainF/Data-Free-Adversarial-Distillation.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• FitNets: Hints for Thin Deep Nets
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• Born Again Neural Networks
• Data-Free Knowledge Distillation for Deep Neural Networks
• Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities

Cited by in corpus (19)

• Contrastive Model Inversion for Data-Free Knowledge Distillation
• Large-Scale Generative Data-Free Distillation
• Distilling Object Detectors via Decoupled Features
• Dataset Inference: Ownership Resolution in Machine Learning
• Source-Free Domain Adaptation for Semantic Segmentation
• Conditional Pseudo-Supervised Contrast for Data-Free Knowledge Distillation
• Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data
• NaturalInversion: Data-Free Image Synthesis Improving Real-World Consistency
• Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compression
• Training Generative Adversarial Networks in One Stage
• Representation Consolidation for Training Expert Students
• Dual Discriminator Adversarial Distillation for Data-free Model Compression
• Adversarial Self-Supervised Data-Free Distillation for Text Classification
• Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation
• Towards Accurate Quantization and Pruning via Data-free Knowledge Transfer
• Zero-shot Adversarial Quantization
• MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing