Secure Distributed Training at Scale

Abstract

Many areas of deep learning benefit from using increasingly larger neural networks trained on public data, as is the case for pre-trained models for NLP and computer vision. Training such models requires a lot of computational resources (e.g., HPC clusters) that are not available to small research groups and independent researchers. One way to address it is for several smaller groups to pool their computational resources together and train a model that benefits all participants. Unfortunately, in this case, any participant can jeopardize the entire training run by sending incorrect updates, deliberately or by mistake. Training in presence of such peers requires specialized distributed training algorithms with Byzantine tolerance. These algorithms often sacrifice efficiency by introducing redundant communication or passing all updates through a trusted server, making it infeasible to apply them to large-scale deep learning, where models can have billions of parameters. In this work, we propose a novel protocol for secure (Byzantine-tolerant) decentralized training that emphasizes communication efficiency.

arXiv commentAccepted to International Conference on Machine Learning (ICML 2022). 61 pages, 10 figures. The version 4 fixes inaccuracies in the proofs of Lemmas E.2 and E.4. Code: https://github.com/yandex-research/btard

References in corpus (23)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Observation of Gravitational Waves from a Binary Black Hole Merger
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Federated Learning for Mobile Keyboard Prediction
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Robust Aggregation for Federated Learning
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Horovod: fast and easy distributed deep learning in TensorFlow
• Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attacks
• Byzantine Stochastic Gradient Descent
• Distributed Learning with Compressed Gradient Differences
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learning
• Linearly Converging Error Compensated SGD
• Learning from History for Byzantine Robust Optimization
• MARINA: Faster Non-Convex Distributed Learning with Compression
• ByGARS: Byzantine SGD with Arbitrary Number of Attackers
• Distributed Deep Learning in Open Collaborations
• Robust Consensus in Distributed Networks using Total Variation