Data-Free Knowledge Distillation for Deep Neural Networks
arXiv:1710.07535
Abstract
Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy. However, all of these approaches rely on access to the original training set, which might not always be possible if the network to be compressed was trained on a very large dataset, or on a dataset whose release poses privacy or safety concerns as may be the case for biometrics tasks. We present a method for data-free knowledge distillation, which is able to compress deep neural networks trained on large-scale datasets to a fraction of their size leveraging only some extra metadata to be provided with a pretrained model release. We also explore different kinds of metadata that can be used with our method, and discuss tradeoffs involved in using each of them.
Accepted to NIPS 2017 Workshop on Learning with Limited Data. Under review at AISTATS 2018
References in corpus (6)
Cited by in corpus (62)
- Knowledge Distillation: A Survey
- Edge Intelligence: Architectures, Challenges, and Applications
- Knowledge Distillation in Deep Learning and its Applications
- Dataset Condensation with Gradient Matching
- Data-Free Adversarial Distillation
- Applications and Techniques for Fast Machine Learning in Science
- And the Bit Goes Down: Revisiting the Quantization of Neural Networks
- Relational Knowledge Distillation
- Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
- Zero-shot Knowledge Transfer via Adversarial Belief Matching
- Contrastive Model Inversion for Data-Free Knowledge Distillation
- Dream Distillation: A Data-Independent Model Compression Framework
- Source Free Domain Adaptation with Image Translation
- Large-Scale Generative Data-Free Distillation
- Adaptive Neural Network-Based Approximation to Accelerate Eulerian Fluid Simulation
- Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression
- Dataset Distillation
- Undistillable: Making A Nasty Teacher That CANNOT teach students
- Few Shot Network Compression via Cross Distillation
- MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks
- Few Sample Knowledge Distillation for Efficient Network Compression
- Synthetic data generation method for data-free knowledge distillation in regression neural networks
- Generative Low-bitwidth Data Free Quantization
- Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model
- Source-Free Domain Adaptation for Semantic Segmentation
- Learnable Boundary Guided Adversarial Training
- A Continual and Incremental Learning Approach for TinyML On-device Training Using Dataset Distillation and Model Size Adaption
- Thief, Beware of What Get You There: Towards Understanding Model Extraction Attack
- Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data
- Revisiting Knowledge Distillation for Object Detection
- Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN
- Data-Free Learning of Student Networks
- Generative Zero-shot Network Quantization
- Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
- Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples
- Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis
- Knowledge Distillation as Semiparametric Inference
- Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model
- Robustness and Diversity Seeking Data-Free Knowledge Distillation
- EdgeAI: A Vision for Deep Learning in IoT Era
- Graph-Free Knowledge Distillation for Graph Neural Networks
- Visualizing Adapted Knowledge in Domain Transfer
- AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compression
- Representation Consolidation for Training Expert Students
- Efficient Encrypted Inference on Ensembles of Decision Trees
- BeSound: Bluetooth-Based Position Estimation Enhancing with Cross-Modality Distillation
- Adversarial Self-Supervised Data-Free Distillation for Text Classification
- Data-free Knowledge Distillation for Segmentation using Data-Enriching GAN
- Dual Discriminator Adversarial Distillation for Data-free Model Compression
- Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation
- Zero-shot Adversarial Quantization
- New Directions in Distributed Deep Learning: Bringing the Network at Forefront of IoT Design
- DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier
- Network-Agnostic Knowledge Transfer for Medical Image Segmentation
- Neighbourhood Distillation: On the benefits of non end-to-end distillation
- Towards Data-Free Domain Generalization
- Impression Space from Deep Template Network
- SlimNets: An Exploration of Deep Model Compression and Acceleration
- On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks
- Towards Zero-Shot Knowledge Distillation for Natural Language Processing
- Filter Grafting for Deep Neural Networks: Reason, Method, and Cultivation
- Knowledge Distillation By Sparse Representation Matching