CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression

Abstract

Due to the high communication cost in distributed and federated learning, methods relying on compressed communication are becoming increasingly popular. Besides, the best theoretically and practically performing gradient-type methods invariably rely on some form of acceleration/momentum to reduce the number of communications (faster convergence), e.g., Nesterov's accelerated gradient descent (Nesterov, 1983, 2004) and Adam (Kingma and Ba, 2014). In order to combine the benefits of communication compression and convergence acceleration, we propose a \emph{compressed and accelerated} gradient method based on ANITA (Li, 2021) for distributed optimization, which we call CANITA. Our CANITA achieves the \emph{first accelerated rate} $O(\sqrt{(1+\sqrt{\frac{{\omega }^3}{n}})\frac{L}{\epsilon }}+\omega (\frac{1}{\epsilon }{)}^{\frac{1}{3}})$, which improves upon the state-of-the-art non-accelerated rate $O\left((1+\frac{\omega }{n})\frac{L}{\epsilon }+\frac{{\omega }^2+\omega }{\omega +n}\frac{1}{\epsilon }\right)$ of DIANA (Khaled et al., 2020) for distributed general convex problems, where $\epsilon$ is the target error, $L$ is the smooth parameter of the objective, $n$ is the number of machines/devices, and $\omega$ is the compression parameter (larger $\omega$ means more compression can be applied, and no compression implies $\omega =0$). Our results show that as long as the number of devices $n$ is large (often true in distributed/federated learning), or the compression $\omega$ is not very high, CANITA achieves the faster convergence rate $O\left(\sqrt{\frac{L}{\epsilon }}\right)$, i.e., the number of communication rounds is $O\left(\sqrt{\frac{L}{\epsilon }}\right)$ (vs. $O\left(\frac{L}{\epsilon }\right)$ achieved by previous works). As a result, CANITA enjoys the advantages of both compression (compressed communication in each round) and acceleration (much fewer communication rounds).

arXiv commentNeurIPS 2021

References in corpus (18)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Sparsified SGD with Memory
• A Universal Catalyst for First-Order Optimization
• Distributed Learning with Compressed Gradient Differences
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• Natural Compression for Distributed Deep Learning
• On Biased Compression for Distributed Learning
• EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback
• Distributed learning with compressed gradients
• An optimal randomized incremental gradient method
• A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization
• ZeroSARAH: Efficient Nonconvex Finite-Sum Optimization with Zero Full Gradient Computation
• Error Compensated Distributed SGD Can Be Accelerated
• ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Method
• A Short Note of PAGE: Optimal Convergence Rates for Nonconvex Optimization
• DEED: A General Quantization Scheme for Communication Efficiency in Bits

Cited by in corpus (2)

• FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learning
• ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Method