A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off

Abstract

Reducing the precision of weights and activation functions in neural network training, with minimal impact on performance, is essential for the deployment of these models in resource-constrained environments. We apply mean-field techniques to networks with quantized activations in order to evaluate the degree to which quantization degrades signal propagation at initialization. We derive initialization schemes which maximize signal propagation in such networks and suggest why this is helpful for generalization. Building on these results, we obtain a closed form implicit equation for $L_{\max }$, the maximal trainable depth (and hence model capacity), given $N$, the number of quantization levels in the activation function. Solving this equation numerically, we obtain asymptotically: $L_{\max }\propto N^{1.82}$.

arXiv commentNIPS 2019

References in corpus (20)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Binarized Neural Networks
• Compressing Neural Networks with the Hashing Trick
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Convolutional Neural Networks using Logarithmic Data Representation
• Exponential expressivity in deep neural networks through transient chaos
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Scalable Methods for 8-bit Training of Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Fixup Initialization: Residual Learning Without Normalization
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• A Mean Field Theory of Batch Normalization
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• The High-Dimensional Geometry of Binary Neural Networks

Cited by in corpus (5)

• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• Finite Versus Infinite Neural Networks: an Empirical Study
• Towards NNGP-guided Neural Architecture Search
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Fractional moment-preserving initialization schemes for training deep neural networks