Bringing Giant Neural Networks Down to Earth with Unlabeled Data

Abstract

Compressing giant neural networks has gained much attention for their extensive applications on edge devices such as cellphones. During the compressing process, one of the most important procedures is to retrain the pre-trained models using the original training dataset. However, due to the consideration of security, privacy or commercial profits, in practice, only a fraction of sample training data are made available, which makes the retraining infeasible. To solve this issue, this paper proposes to resort to unlabeled data in hand that can be cheaper to acquire. Specifically, we exploit the unlabeled data to mimic the classification characteristics of giant networks, so that the original capacity can be preserved nicely. Nevertheless, there exists a dataset bias between the labeled and unlabeled data, which may disturb the training and degrade the performance. We thus fix this bias by an adversarial loss to make an alignment on the distributions of their low-level feature representations. We further provide theoretical discussions about how the unlabeled data help compressed networks to generalize better. Experimental results demonstrate that the unlabeled data can significantly improve the performance of the compressed networks.

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Unsupervised Domain Adaptation with Residual Transfer Networks
• Rethinking the Value of Network Pruning
• Binarized Neural Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• Pruning Filters for Efficient ConvNets
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Auxiliary Deep Generative Models
• Dynamic Channel Pruning: Feature Boosting and Suppression
• Learning the Number of Neurons in Deep Networks
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Compression-aware Training of Deep Networks
• One-Shot Generalization in Deep Generative Models
• Deep Predictive Coding Network for Object Recognition

Cited by in corpus (5)

• SCOP: Scientific Control for Reliable Neural Network Pruning
• Locally Free Weight Sharing for Network Width Search
• GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet
• A novel method for identifying the deep neural network model with the Serial Number
• BCNet: Searching for Network Width with Bilaterally Coupled Network