Distributed Deep Learning in Open Collaborations
arXiv:2106.10207
Abstract
Modern deep learning applications require increasingly more compute to train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High-Performance Computing clusters, whose construction and maintenance are both environmentally costly and well beyond the budget of most organizations. As a result, some research directions become the exclusive domain of a few large industrial and even fewer academic actors. To alleviate this disparity, smaller groups may pool their computational resources and run collaborative experiments that benefit all participants. This paradigm, known as grid- or volunteer computing, has seen successful applications in numerous scientific areas. However, using this approach for machine learning is difficult due to high latency, asymmetric bandwidth, and several challenges unique to volunteer computing. In this work, we carefully analyze these constraints and propose a novel algorithmic framework designed specifically for collaborative training. We demonstrate the effectiveness of our approach for SwAV and ALBERT pretraining in realistic conditions and achieve performance comparable to traditional setups at a fraction of the cost. Finally, we provide a detailed report of successful collaborative language model pretraining with 40 participants.
arXiv commentAccepted to Conference on Neural Information Processing Systems (NeurIPS) 2021. 32 pages, 10 figures. Code: https://github.com/yandex-research/DeDLOC
References in corpus (28)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Scaling Laws for Neural Language Models
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Horovod: fast and easy distributed deep learning in TensorFlow
- Large Batch Training of Convolutional Networks
- Pointer Sentinel Mixture Models
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Peer-to-Peer Communication Across Network Address Translators
- Training Tips for the Transformer Model
- Folding@Home and Genome@Home: Using distributed computing to tackle previously intractable problems in computational biology
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery
- Transfer Learning in Deep Reinforcement Learning: A Survey
- Stochastic Gradient Push for Distributed Deep Learning
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models
- Federated Learning with Differential Privacy: Algorithms and Performance Analysis
- Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
- ZeRO-Offload: Democratizing Billion-Scale Model Training
- Unified Optimal Analysis of the (Stochastic) Gradient Method
- GLU Variants Improve Transformer
- Multi-node Bert-pretraining: Cost-efficient Approach
- Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices