Zero-Shot Knowledge Distillation in Deep Networks

Abstract

Knowledge distillation deals with the problem of training a smaller model (Student) from a high capacity source model (Teacher) so as to retain most of its performance. Existing approaches use either the training data or meta-data extracted from it in order to train the Student. However, accessing the dataset on which the Teacher has been trained may not always be feasible if the dataset is very large or it poses privacy or safety concerns (e.g., bio-metric or medical data). Hence, in this paper, we propose a novel data-free method to train the Student from the Teacher. Without even using any meta-data, we synthesize the Data Impressions from the complex Teacher model and utilize these as surrogates for the original training data samples to transfer its learning to Student via knowledge distillation. We, therefore, dub our method "Zero-Shot Knowledge Distillation" and demonstrate that our framework results in competitive generalization performance as achieved by distillation using the actual training data samples on multiple benchmark datasets.

Accepted in ICML 2019, codes will be available at https://github.com/vcl-iisc/ZSKD

Cited by in corpus (21)

• Adversarial Examples Make Strong Poisons
• Large-Scale Generative Data-Free Distillation
• Why distillation helps: a statistical perspective
• Source-Free Domain Adaptation for Semantic Segmentation
• Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup
• Generative Zero-shot Network Quantization
• Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples
• Robustness and Diversity Seeking Data-Free Knowledge Distillation
• Representation Consolidation for Training Expert Students
• Efficient Encrypted Inference on Ensembles of Decision Trees
• Domain Impression: A Source Data Free Domain Adaptation Method
• FedRAD: Federated Robust Adaptive Distillation
• DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier
• Dynamic Knowledge Distillation for Black-box Hypothesis Transfer Learning
• Zero-shot Adversarial Quantization
• Data-free Knowledge Distillation for Segmentation using Data-Enriching GAN
• Towards Zero-Shot Knowledge Distillation for Natural Language Processing
• MINIMAL: Mining Models for Data Free Universal Adversarial Triggers
• ZS-IL: Looking Back on Learned Experiences For Zero-Shot Incremental Learning
• Differential Replication in Machine Learning
• Unlabeled Data Deployment for Classification of Diabetic Retinopathy Images Using Knowledge Transfer