Moniqua: Modulo Quantized Communication in Decentralized SGD

Abstract

Running Stochastic Gradient Descent (SGD) in a decentralized fashion has shown promising results. In this paper we propose Moniqua, a technique that allows decentralized SGD to use quantized communication. We prove in theory that Moniqua communicates a provably bounded number of bits per iteration, while converging at the same asymptotic rate as the original algorithm does with full-precision communication. Moniqua improves upon prior works in that it (1) requires zero additional memory, (2) works with 1-bit quantization, and (3) is applicable to a variety of decentralized algorithms. We demonstrate empirically that Moniqua converges faster with respect to wall clock time than other quantized decentralized algorithms. We also show that Moniqua is robust to very low bit-budgets, allowing 1-bit-per-parameter communication without compromising validation accuracy when training ResNet20 and ResNet110 on CIFAR10.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Sparsified SGD with Memory
• Local SGD Converges Fast and Communicates Little
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Communication Compression for Decentralized Training
• D${}^2$: Decentralized Training over Decentralized Data
• Distributed Mean Estimation with Limited Communication
• Stochastic Gradient Push for Distributed Deep Learning
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• High-Accuracy Low-Precision Training
• Distributed Learning over Unreliable Networks
• Distributed Learning with Sublinear Communication
• Accelerated Decentralized Optimization with Local Updates for Smooth and Strongly Convex Objectives

Cited by in corpus (10)

• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk Minimization
• Fully Asynchronous Distributed Optimization with Linear Convergence in Directed Networks
• Linear Convergent Decentralized Optimization with Compression
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
• Innovation Compression for Communication-efficient Distributed Optimization with Linear Convergence
• New Bounds For Distributed Mean Estimation and Variance Reduction
• Cross-Gradient Aggregation for Decentralized Learning from Non-IID data
• Optimal Complexity in Decentralized Training
• Communication Efficient Generalized Tensor Factorization for Decentralized Healthcare Networks