8-Bit Approximations for Parallelism in Deep Learning

Abstract

The creation of practical deep learning data-products often requires parallelization across processors and computers to make deep learning feasible on large data sets, but bottlenecks in communication bandwidth make it difficult to attain good speedups through parallelism. Here we develop and test 8-bit approximation algorithms which make better use of the available bandwidth by compressing 32-bit gradients and nonlinear activations to 8-bit approximations. We show that these approximations do not decrease predictive performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism and provide a data transfer speedup of 2x relative to 32-bit parallelism. We build a predictive model for speedups based on our experimental data, verify its validity on known speedup data, and show that we can obtain a speedup of 50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We compare our data types with other methods and show that 8-bit approximations achieve state-of-the-art speedups for model parallelism. Thus 8-bit approximation is an efficient method to parallelize convolutional networks on very large systems of GPUs.

References in corpus (4)

• Deep Learning in Neural Networks: An Overview
• Deep Learning with Limited Numerical Precision
• One weird trick for parallelizing convolutional neural networks
• Deep Image: Scaling up Image Recognition

Cited by in corpus (21)

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Rethinking floating point for deep learning
• A Study of BFLOAT16 for Deep Learning Training
• FedLab: A Flexible Federated Learning Framework
• Ternary Neural Networks with Fine-Grained Quantization
• FLrce: Resource-Efficient Federated Learning with Early-Stopping Strategy
• On the Utility of Gradient Compression in Distributed Training Systems
• On Scale-out Deep Learning Training for Cloud and HPC
• Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Boosting Distributed Machine Learning Training Through Loss-tolerant Transmission Protocol
• 8-bit Optimizers via Block-wise Quantization
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• A Very Compact Embedded CNN Processor Design Based on Logarithmic Computing
• A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
• MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training
• Deep Neural Network Training without Multiplications
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research
• Full-stack Optimization for Accelerating CNNs with FPGA Validation
• Fixed-point Factorized Networks