Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantization

Abstract

Generative data-free quantization emerges as a practical compression approach that quantizes deep neural networks to low bit-width without accessing the real data. This approach generates data utilizing batch normalization (BN) statistics of the full-precision networks to quantize the networks. However, it always faces the serious challenges of accuracy degradation in practice. We first give a theoretical analysis that the diversity of synthetic samples is crucial for the data-free quantization, while in existing approaches, the synthetic data completely constrained by BN statistics experimentally exhibit severe homogenization at distribution and sample levels. This paper presents a generic Diverse Sample Generation (DSG) scheme for the generative data-free quantization, to mitigate detrimental homogenization. We first slack the statistics alignment for features in the BN layer to relax the distribution constraint. Then, we strengthen the loss impact of the specific BN layers for different samples and inhibit the correlation among samples in the generation process, to diversify samples from the statistical and spatial perspectives, respectively. Comprehensive experiments show that for large-scale image classification tasks, our DSG can consistently quantization performance on different neural architectures, especially under ultra-low bit-width. And data diversification caused by our DSG brings a general gain to various quantization-aware training and post-training quantization approaches, demonstrating its generality and effectiveness.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Binary Neural Networks: A Survey
• Discrimination-aware Network Pruning for Deep Model Compression
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Resiliency of Deep Neural Networks under Quantization
• Post-training 4-bit quantization of convolution networks for rapid-deployment
• Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity
• Hierarchical binary CNNs for landmark localization with limited resources
• Generative Zero-shot Network Quantization