signSGD with Majority Vote is Communication Efficient And Fault Tolerant

Abstract

Training neural networks on large datasets can be accelerated by distributing the workload over a network of machines. As datasets grow ever larger, networks of hundreds or thousands of machines become economically viable. The time cost of communicating gradients limits the effectiveness of using such large machine counts, as may the increased chance of network faults. We explore a particularly simple algorithm for robust, communication-efficient learning---signSGD. Workers transmit only the sign of their gradient vector to a server, and the overall update is decided by a majority vote. This algorithm uses $32\times$ less communication per iteration than full-precision, distributed SGD. Under natural conditions verified by experiment, we prove that signSGD converges in the large and mini-batch settings, establishing convergence for a parameter regime of Adam as a byproduct. Aggregating sign gradients by majority vote means that no individual worker has too much power. We prove that unlike SGD, majority vote is robust when up to 50% of workers behave adversarially. The class of adversaries we consider includes as special cases those that invert or randomise their gradient estimate. On the practical side, we built our distributed training system in Pytorch. Benchmarking against the state of the art collective communications library (NCCL), our framework---with the parameter server housed entirely on one machine---led to a 25% reduction in time for training resnet50 on Imagenet when using 15 AWS p3.2xlarge machines.

Cited by in corpus (30)

• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Robust Federated Learning with Noisy Labels
• Variance Reduced Local SGD with Lower Communication Complexity
• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
• Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
• Federated and distributed learning applications for electronic health records and structured medical data: A scoping review
• A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learning
• Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms
• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks
• On the Utility of Gradient Compression in Distributed Training Systems
• Byzantines can also Learn from History: Fall of Centered Clipping in Federated Learning
• Distributed Momentum for Byzantine-resilient Learning
• Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
• BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning
• Byzantine Fault-Tolerant Distributed Machine Learning Using Stochastic Gradient Descent (SGD) and Norm-Based Comparative Gradient Elimination (CGE)
• Stochastic Sign Descent Methods: New Algorithms and Better Theory
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• The Geometry of Sign Gradient Descent
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
• Approximate Byzantine Fault-Tolerance in Distributed Optimization
• Communication-Efficient Distributed SGD with Error-Feedback, Revisited
• Pufferfish: Communication-efficient Models At No Extra Cost
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
• Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
• On Faster Convergence of Scaled Sign Gradient Descent
• ByzShield: An Efficient and Robust System for Distributed Training
• Distributed Sparse SGD with Majority Voting
• On The Convergence of Euler Discretization of Finite-Time Convergent Gradient Flows
• Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients