Zero-shot Knowledge Transfer via Adversarial Belief Matching

Abstract

Performing knowledge transfer from a large teacher network to a smaller student is a popular task in modern deep learning applications. However, due to growing dataset sizes and stricter privacy regulations, it is increasingly common not to have access to the data that was used to train the teacher. We propose a novel method which trains a student to match the predictions of its teacher without using any data or metadata. We achieve this by training an adversarial generator to search for images on which the student poorly matches the teacher, and then using them to train the student. Our resulting student closely approximates its teacher for simple datasets like SVHN, and on CIFAR10 we improve on the state-of-the-art for few-shot distillation (with 100 images per class), despite using no data. Finally, we also propose a metric to quantify the degree of belief matching between teacher and student in the vicinity of decision boundaries, and observe a significantly higher match between our zero-shot student and the teacher, than between a student distilled with real data and the teacher. Code available at: https://github.com/polo5/ZeroShotKnowledgeTransfer

References in corpus (21)

• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Residual Learning for Image Recognition
• WaveNet: A Generative Model for Raw Audio
• Wide Residual Networks
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Deep Learning with Limited Numerical Precision
• Stealing Machine Learning Models via Prediction APIs
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Adversarial Examples Are Not Bugs, They Are Features
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Data-Free Knowledge Distillation for Deep Neural Networks
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Moonshine: Distilling with Cheap Convolutions
• Zero-Shot Knowledge Distillation in Deep Networks
• Policy Distillation
• Few-shot learning of neural networks from scratch by pseudo example optimization
• Variational Information Distillation for Knowledge Transfer
• Dataset Distillation

Cited by in corpus (5)

• Knowledge Distillation: A Survey
• High Accuracy and High Fidelity Extraction of Neural Networks
• Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data
• Robustness and Diversity Seeking Data-Free Knowledge Distillation
• Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation