Dream Distillation: A Data-Independent Model Compression Framework

Abstract

Model compression is eminently suited for deploying deep learning on IoT-devices. However, existing model compression techniques rely on access to the original or some alternate dataset. In this paper, we address the model compression problem when no real data is available, e.g., when data is private. To this end, we propose Dream Distillation, a data-independent model compression framework. Our experiments show that Dream Distillation can achieve 88.5% accuracy on the CIFAR-10 test set without actually training on the original data!

Presented at the ICML 2019 Joint Workshop on On-Device Machine Learning & Compact Deep Neural Network Representations (ODML-CDNNR)

References in corpus (3)

• Distilling the Knowledge in a Neural Network
• Data-Free Knowledge Distillation for Deep Neural Networks
• Knowledge distillation using unlabeled mismatched images

Cited by in corpus (7)

• Data from Model: Extracting Data from Non-robust and Robust Models
• Half-Real Half-Fake Distillation for Class-Incremental Semantic Segmentation
• Generative Zero-shot Network Quantization
• EdgeAI: A Vision for Deep Learning in IoT Era
• Robustness and Diversity Seeking Data-Free Knowledge Distillation
• Representation Consolidation for Training Expert Students
• Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation