MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization
arXiv:2106.03056
Abstract
We propose a generic variance-reduced algorithm, which we call MUltiple RANdomized Algorithm (MURANA), for minimizing a sum of several smooth functions plus a regularizer, in a sequential or distributed manner. Our method is formulated with general stochastic operators, which allow us to model various strategies for reducing the computational complexity. For example, MURANA supports sparse activation of the gradients, and also reduction of the communication load via compression of the update vectors. This versatility allows MURANA to cover many existing randomization mechanisms within a unified framework, which also makes it possible to design new methods as special cases.
3rd Annual Conference on Mathematical and Scientific Machine Learning (MSML), Aug. 2022. PMLR 190:81-96
References in corpus (10)
- Federated Learning: Strategies for Improving Communication Efficiency
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
- Distributed Learning with Compressed Gradient Differences
- Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
- Optimal Gradient Compression for Distributed and Federated Learning
- Linearly Converging Error Compensated SGD
- A Double Residual Compression Algorithm for Efficient Distributed Learning
- Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees