Zero-Shot Knowledge Distillation in Deep Networks

Abstract

Knowledge distillation deals with the problem of training a smaller model (Student) from a high capacity source model (Teacher) so as to retain most of its performance. Existing approaches use either the training data or meta-data extracted from it in order to train the Student. However, accessing the dataset on which the Teacher has been trained may not always be feasible if the dataset is very large or it poses privacy or safety concerns (e.g., bio-metric or medical data). Hence, in this paper, we propose a novel data-free method to train the Student from the Teacher. Without even using any meta-data, we synthesize the Data Impressions from the complex Teacher model and utilize these as surrogates for the original training data samples to transfer its learning to Student via knowledge distillation. We, therefore, dub our method "Zero-Shot Knowledge Distillation" and demonstrate that our framework results in competitive generalization performance as achieved by distillation using the actual training data samples on multiple benchmark datasets.

Accepted in ICML 2019, codes will be available at https://github.com/vcl-iisc/ZSKD

Cited by in corpus (37)

• Knowledge Distillation in Deep Learning and its Applications
• Dataset Condensation with Gradient Matching
• Self-Distillation Amplifies Regularization in Hilbert Space
• Applications and Techniques for Fast Machine Learning in Science
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Adversarial Examples Make Strong Poisons
• Source Free Domain Adaptation with Image Translation
• Large-Scale Generative Data-Free Distillation
• Black-Box Ripper: Copying black-box models using generative evolutionary algorithms
• Why distillation helps: a statistical perspective
• Generative Low-bitwidth Data Free Quantization
• Source-Free Domain Adaptation for Semantic Segmentation
• A Meta-Learning Framework for Generalized Zero-Shot Learning
• Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• Generative Zero-shot Network Quantization
• Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples
• Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model
• Robustness and Diversity Seeking Data-Free Knowledge Distillation
• Graph-Free Knowledge Distillation for Graph Neural Networks
• Visualizing Adapted Knowledge in Domain Transfer
• Representation Consolidation for Training Expert Students
• Efficient Encrypted Inference on Ensembles of Decision Trees
• Domain Impression: A Source Data Free Domain Adaptation Method
• DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier
• Dual Discriminator Adversarial Distillation for Data-free Model Compression
• FedRAD: Federated Robust Adaptive Distillation
• Towards Accurate Quantization and Pruning via Data-free Knowledge Transfer
• Data-free Knowledge Distillation for Segmentation using Data-Enriching GAN
• Zero-shot Adversarial Quantization
• Dynamic Knowledge Distillation for Black-box Hypothesis Transfer Learning
• Towards Zero-Shot Knowledge Distillation for Natural Language Processing
• ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning
• Unlabeled Data Deployment for Classification of Diabetic Retinopathy Images Using Knowledge Transfer
• Towards Data-Free Domain Generalization
• MINIMAL: Mining Models for Data Free Universal Adversarial Triggers
• Differential Replication in Machine Learning