Teacher-Student Compression with Generative Adversarial Networks

Abstract

More accurate machine learning models often demand more computation and memory at test time, making them difficult to deploy on CPU- or memory-constrained devices. Teacher-student compression (TSC), also known as distillation, alleviates this burden by training a less expensive student model to mimic the expensive teacher model while maintaining most of the original accuracy. However, when fresh data is unavailable for the compression task, the teacher's training data is typically reused, leading to suboptimal compression. In this work, we propose to augment the compression dataset with synthetic data from a generative adversarial network (GAN) designed to approximate the training data distribution. Our GAN-assisted TSC (GAN-TSC) significantly improves student accuracy for expensive models such as large random forests and deep neural networks on both tabular and image datasets. Building on these results, we propose a comprehensive metric---the TSC Score---to evaluate the quality of synthetic datasets based on their induced TSC performance. The TSC Score captures both data diversity and class affinity, and we illustrate its benefits over the popular Inception Score in the context of image classification.

References in corpus (13)

• Scikit-learn: Machine Learning in Python
• Distilling the Knowledge in a Neural Network
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Do Deep Nets Really Need to be Deep?
• Compressing Neural Networks with the Hashing Trick
• A note on the evaluation of generative models
• Generating Multi-label Discrete Patient Records using Generative Adversarial Networks
• GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
• Distilling a Neural Network Into a Soft Decision Tree
• MLBench: How Good Are Machine Learning Clouds for Binary Classification Tasks on Structured Data?
• Lossless (and Lossy) Compression of Random Forests

Cited by in corpus (6)

• Knowledge Distillation: A Survey
• Classification Accuracy Score for Conditional Generative Models
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• Knowledge Distillation as Semiparametric Inference
• Efficient Encrypted Inference on Ensembles of Decision Trees
• Locally Linear Region Knowledge Distillation