Training Quantized Nets: A Deeper Understanding

Abstract

Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towards learning on embedded platforms that have limited computing resources, memory capacity, and power consumption. Numerous recent publications have studied methods for training quantized networks, but these studies have mostly been empirical. In this work, we investigate training methods for quantized neural networks from a theoretical viewpoint. We first explore accuracy guarantees for training methods under convexity assumptions. We then look at the behavior of these algorithms for non-convex problems, and show that training algorithms that exploit high-precision representations have an important greedy search phase that purely quantized training methods lack, which explains the difficulty of training using low-precision arithmetic.

NIPS 2017

References in corpus (7)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Ternary Weight Networks
• Convolutional Neural Networks using Logarithmic Data Representation
• Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation

Cited by in corpus (14)

• Review: Deep Learning in Electron Microscopy
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs
• Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering
• Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• Universal Approximation Theorems of Fully Connected Binarized Neural Networks
• A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
• Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1