The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning

Abstract

Recently there has been significant interest in training machine-learning models at low precision: by reducing precision, one can reduce computation and communication by one order of magnitude. We examine training at reduced precision, both from a theoretical and practical perspective, and ask: is it possible to train models at end-to-end low precision with provable guarantees? Can this lead to consistent order-of-magnitude speedups? We present a framework called ZipML to answer these questions. For linear models, the answer is yes. We develop a simple framework based on one simple but novel strategy called double sampling. Our framework is able to execute training at low precision with no bias, guaranteeing convergence, whereas naive quantization would introduce significant bias. We validate our framework across a range of applications, and show that it enables an FPGA prototype that is up to 6.5x faster than an implementation using full 32-bit precision. We further develop a variance-optimal stochastic quantization strategy and show that it can make a significant difference in a variety of settings. When applied to linear models together with double sampling, we save up to another 1.7x in data movement compared with uniform quantization. When training deep networks with quantized models, we achieve higher accuracy than the state-of-the-art XNOR-Net. Finally, we extend our framework through approximation to non-linear models, such as SVM. We show that, although using low-precision data induces bias, we can appropriately bound and control the bias. We find in practice 8-bit precision is often sufficient to converge to the correct solution. Interestingly, however, in practice we notice that our framework does not always outperform the naive rounding approach. We discuss this negative result in detail.

References in corpus (12)

• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Ternary Weight Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Convolutional Neural Networks using Logarithmic Data Representation
• Bitwise Neural Networks
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Principal Component Projection Without Principal Component Analysis
• Faster Principal Component Regression and Stable Matrix Chebyshev Approximation

Cited by in corpus (7)

• Gradient Sparsification for Communication-Efficient Distributed Optimization
• High-Accuracy Low-Precision Training
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Low-Memory Neural Network Training: A Technical Report
• Communication trade-offs for synchronized distributed SGD with large step size
• Reproducible Floating-Point Aggregation in RDBMSs
• How Low Can We Go: Trading Memory for Error in Low-Precision Training