MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing

Abstract

User data confidentiality protection is becoming a rising challenge in the present deep learning research. Without access to data, conventional data-driven model compression faces a higher risk of performance degradation. Recently, some works propose to generate images from a specific pretrained model to serve as training data. However, the inversion process only utilizes biased feature statistics stored in one model and is from low-dimension to high-dimension. As a consequence, it inevitably encounters the difficulties of generalizability and inexact inversion, which leads to unsatisfactory performance. To address these problems, we propose MixMix based on two simple yet effective techniques: (1) Feature Mixing: utilizes various models to construct a universal feature space for generalized inversion; (2) Data Mixing: mixes the synthesized images and labels to generate exact label information. We prove the effectiveness of MixMix from both theoretical and empirical perspectives. Extensive experiments show that MixMix outperforms existing methods on the mainstream compression tasks, including quantization, knowledge distillation, and pruning. Specifically, MixMix achieves up to 4% and 20% accuracy uplift on quantization and pruning, respectively, compared to existing data-free compression work.

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Trained Ternary Quantization
• Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images
• Model compression via distillation and quantization
• Faster gaze prediction with dense networks and Fisher pruning
• The Expressive Power of Neural Networks: A View from the Width
• Data-Free Adversarial Distillation
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• Up or Down? Adaptive Rounding for Post-Training Quantization
• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning