SWALP : Stochastic Weight Averaging in Low-Precision Training

Abstract

Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.

arXiv commentPublished at ICML 2019

Cited by in corpus (6)

• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Glyph: Fast and Accurately Training Deep Neural Networks on Encrypted Data
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR