Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model

Abstract

Knowledge distillation (KD) is a successful approach for deep neural network acceleration, with which a compact network (student) is trained by mimicking the softmax output of a pre-trained high-capacity network (teacher). In tradition, KD usually relies on access to the training samples and the parameters of the white-box teacher to acquire the transferred knowledge. However, these prerequisites are not always realistic due to storage costs or privacy issues in real-world applications. Here we propose the concept of decision-based black-box (DB3) knowledge distillation, with which the student is trained by distilling the knowledge from a black-box teacher (parameters are not accessible) that only returns classes rather than softmax outputs. We start with the scenario when the training set is accessible. We represent a sample's robustness against other classes by computing its distances to the teacher's decision boundaries and use it to construct the soft label for each training sample. After that, the student can be trained via standard KD. We then extend this approach to a more challenging scenario in which even accessing the training data is not feasible. We propose to generate pseudo samples distinguished by the teacher's decision boundaries to the largest extent and construct soft labels for them, which are used as the transfer set. We evaluate our approaches on various benchmark networks and datasets and experiment results demonstrate their effectiveness. Codes are available at: https://github.com/zwang84/zsdb3kd.

arXiv commentAccepted to ICML 2021

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Pruning Filters for Efficient ConvNets
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Born Again Neural Networks
• Data-Free Knowledge Distillation for Deep Neural Networks
• Towards Understanding Knowledge Distillation
• Query-Efficient Hard-label Black-box Attack:An Optimization-based Approach
• Paraphrasing Complex Network: Network Compression via Factor Transfer
• Stochastic Zeroth-order Optimization in High Dimensions
• Sign-OPT: A Query-Efficient Hard-label Adversarial Attack
• A geometry-inspired decision-based attack
• Uncertainty-Aware Multi-Shot Knowledge Distillation for Image-Based Object Re-Identification
• Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis