BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning

Abstract

Decentralized algorithm is a form of computation that achieves a global goal through local dynamics that relies on low-cost communication between directly-connected agents. On large-scale optimization tasks involving distributed datasets, decentralized algorithms have shown strong, sometimes superior, performance over distributed algorithms with a central node. Recently, developing decentralized algorithms for deep learning has attracted great attention. They are considered as low-communication-overhead alternatives to those using a parameter server or the Ring-Allreduce protocol. However, the lack of an easy-to-use and efficient software package has kept most decentralized algorithms merely on paper. To fill the gap, we introduce BlueFog, a python library for straightforward, high-performance implementations of diverse decentralized algorithms. Based on a unified abstraction of various communication operations, BlueFog offers intuitive interfaces to implement a spectrum of decentralized algorithms, from those using a static, undirected graph for synchronous operations to those using dynamic and directed graphs for asynchronous operations. BlueFog also adopts several system-level acceleration techniques to further optimize the performance on the deep learning tasks. On mainstream DNN training tasks, BlueFog reaches a much higher throughput and achieves an overall $1.2\times \sim 1.8\times$ speedup over Horovod, a state-of-the-art distributed deep learning package based on Ring-Allreduce. BlueFog is open source at https://github.com/Bluefog-Lib/bluefog.

References in corpus (24)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• On the Linear Convergence of the ADMM in Decentralized Consensus Optimization
• Diffusion Adaptation Strategies for Distributed Optimization and Learning over Networks
• Horovod: fast and easy distributed deep learning in TensorFlow
• A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates
• Optimal algorithms for smooth and strongly convex distributed optimization in networks
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• D${}^2$: Decentralized Training over Decentralized Data
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Stochastic Gradient Push for Distributed Deep Learning
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• An improved convergence analysis for decentralized online stochastic non-convex optimization
• A Unified and Refined Convergence Analysis for Non-Convex Decentralized Learning
• Consensus Control for Decentralized Deep Learning
• Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD
• RelaySum for Decentralized Deep Learning on Heterogeneous Data
• A Stochastic Proximal Gradient Framework for Decentralized Non-Convex Composite Optimization: Topology-Independent Sample Complexity and Communication Efficiency
• Exponential Graph is Provably Efficient for Decentralized Deep Training
• Improving the Transient Times for Distributed Stochastic Gradient Methods
• DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training

Cited by in corpus (5)

• A Simple Python Testbed for Federated Learning Algorithms
• Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD
• Developing Elementary Federated Learning Algorithms Leveraging the ChatGPT
• Heavy-Tail Phenomenon in Decentralized SGD
• ${\text{A}}^2{\text{CiD}}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning