Horovod: fast and easy distributed deep learning in TensorFlow
arXiv:1802.05799
Abstract
Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complications. First, the training library must support inter-GPU communication. Depending on the particular methods employed, this communication may entail anywhere from negligible to significant overhead. Second, the user must modify his or her training code to take advantage of inter-GPU communication. Depending on the training library's API, the modification required may be either significant or minimal. Existing methods for enabling multi-GPU training under the TensorFlow library entail non-negligible communication overhead and require users to heavily modify their model-building code, leading many researchers to avoid the whole mess and stick with slower single-GPU training. In this paper we introduce Horovod, an open source library that improves on both obstructions to scaling: it employs efficient inter-GPU communication via ring reduction and requires only a few lines of modification to user code, enabling faster, easier distributed training in TensorFlow. Horovod is available under the Apache 2.0 license at https://github.com/uber/horovod
References in corpus (3)
Cited by in corpus (191)
- ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data
- A Survey on Distributed Machine Learning
- DeePMD-kit v2: A software package for Deep Potential models
- Self-Supervised Graph Transformer on Large-Scale Molecular Data
- FedML: A Research Library and Benchmark for Federated Machine Learning
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Ray: A Distributed Framework for Emerging AI Applications
- FinBERT: A Pretrained Language Model for Financial Communications
- Scaling Distributed Machine Learning with In-Network Aggregation
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
- NEZHA: Neural Contextualized Representation for Chinese Language Understanding
- Predicting orientation-dependent plastic susceptibility from static structure in amorphous solids via deep learning
- Non-Intrusive Reduced-Order Modeling Using Uncertainty-Aware Deep Neural Networks and Proper Orthogonal Decomposition: Application to Flood Modeling
- ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
- Communication optimization strategies for distributed deep neural network training: A survey
- Understanding Top-k Sparsification in Distributed Deep Learning
- Accelerating Finite-temperature Kohn-Sham Density Functional Theory with Deep Neural Networks
- The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020
- Convergence of Artificial Intelligence and High Performance Computing on NSF-supported Cyberinfrastructure
- Hierarchical, rotation-equivariant neural networks to select structural models of protein complexes
- Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
- Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
- Augment your batch: better training with larger batches
- Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
- Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation
- Flare: Flexible In-Network Allreduce
- TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism
- Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
- Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
- Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving
- DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
- AI in Human-computer Gaming: Techniques, Challenges and Opportunities
- OneFlow: Redesign the Distributed Deep Learning Framework from Scratch
- Hierarchical Federated Learning through LAN-WAN Orchestration
- An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
- Exascale Deep Learning for Scientific Inverse Problems
- DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
- Chainer: A Deep Learning Framework for Accelerating the Research Cycle
- Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks
- Communication-Efficient Edge AI: Algorithms and Systems
- Deep Leakage from Gradients
- Mastering Complex Control in MOBA Games with Deep Reinforcement Learning
- Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
- DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs
- Margin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognition
- TF-Replicator: Distributed Machine Learning for Researchers
- Convolutional Neural Networks for cytoarchitectonic brain mapping at large scale
- Cloudy with high chance of DBMS: A 10-year prediction for Enterprise-Grade ML
- Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
- HeAT -- a Distributed and GPU-accelerated Tensor Framework for Data Analytics
- On the Utility of Gradient Compression in Distributed Training Systems
- C2FNAS: Coarse-to-Fine Neural Architecture Search for 3D Medical Image Segmentation
- USID and Pycroscopy -- Open frameworks for storing and analyzing spectroscopic and imaging data
- UPSNet: A Unified Panoptic Segmentation Network
- FanStore: Enabling Efficient and Scalable I/O for Distributed Deep Learning
- PyKaldi2: Yet another speech toolkit based on Kaldi and PyTorch
- GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
- Highly-scalable, physics-informed GANs for learning solutions of stochastic PDEs
- Hardware/Software Co-Exploration of Neural Architectures
- Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
- Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
- Distributed Learning of Deep Neural Networks using Independent Subnet Training
- Union: An Automatic Workload Manager for Accelerating Network Simulation
- PolyTransform: Deep Polygon Transformer for Instance Segmentation
- HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
- Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
- NetReduce: RDMA-Compatible In-Network Reduction for Distributed DNN Training Acceleration
- TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning
- Domain-specific Communication Optimization for Distributed DNN Training
- CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
- On the Impact of Device and Behavioral Heterogeneity in Federated Learning
- BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning
- Deep Learning Hamiltonian Monte Carlo
- Graph Generative Models for Fast Detector Simulations in High Energy Physics
- Fully Decoupled Neural Network Learning Using Delayed Gradients
- A Tensor Compiler for Unified Machine Learning Prediction Serving
- KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
- Training Multilingual Pre-trained Language Model with Byte-level Subwords
- SWAT Watershed Model Calibration using Deep Learning
- Extracting Variable-Depth Logical Document Hierarchy from Long Documents: Method, Evaluation, and Application
- end-to-end training of a large vocabulary end-to-end speech recognition system
- Performance Analysis and Comparison of Distributed Machine Learning Systems
- Parallelizing Machine Learning as a Service for the End-User
- Parallel Training of Deep Networks with Local Updates
- A Distributed Synchronous SGD Algorithm with Global Top- Sparsification for Low Bandwidth Networks
- Improved particle-flow event reconstruction with scalable neural networks for current and future particle detectors
- Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos
- Temporal Context Aggregation for Video Retrieval with Contrastive Learning
- Czert -- Czech BERT-like Model for Language Representation
- Communication-efficient SGD: From Local SGD to One-Shot Averaging
- Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Training
- Heterogeneity-Aware Asynchronous Decentralized Training
- Federated Transfer Learning with Dynamic Gradient Aggregation
- High-Performance Deep Learning via a Single Building Block
- Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement
- Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning
- Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)
- A deep learning modeling framework to capture mixing patterns in reactive-transport systems
- Emergent Road Rules In Multi-Agent Driving Environments
- Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent
- NeurST: Neural Speech Translation Toolkit
- Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
- Bayesian Neural Networks at Scale: A Performance Analysis and Pruning Study
- DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning
- SceneGen: Learning to Generate Realistic Traffic Scenes
- Scalable Deep-Learning-Accelerated Topology Optimization for Additively Manufactured Materials
- EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
- A Comparative Study of Machine Learning Models for Tabular Data Through Challenge of Monitoring Parkinson's Disease Progression Using Voice Recordings
- AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data
- Protein model quality assessment using rotation-equivariant, hierarchical neural networks
- Addressing Class Imbalance in Federated Learning
- Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods
- NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
- Is Network the Bottleneck of Distributed Training?
- Democratizing Production-Scale Distributed Deep Learning
- S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
- HyPar-Flow: Exploiting MPI and Keras for Scalable Hybrid-Parallel DNN Training using TensorFlow
- Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent
- VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware
- Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers
- Efficient and Less Centralized Federated Learning
- Distributed Deep Learning in Open Collaborations
- Stanza: Layer Separation for Distributed Training in Deep Learning
- A Study of Checkpointing in Large Scale Training of Deep Neural Networks
- COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression
- Dynamic Scheduling of MPI-based Distributed Deep Learning Training Jobs
- Compressing gradients by exploiting temporal correlation in momentum-SGD
- Deep learning at scale for subgrid modeling in turbulent flows
- Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
- Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
- Tensor Relational Algebra for Machine Learning System Design
- Hydra: A Peer to Peer Distributed Training & Data Collection Framework
- User Embedding based Neighborhood Aggregation Method for Inductive Recommendation
- Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training
- Deep Fusion Network for Image Completion
- Looking for change? Roll the Dice and demand Attention
- Crossover-SGD: A gossip-based communication in distributed deep learning for alleviating large mini-batch problem and enhancing scalability
- Application of variational policy gradient to atomic-scale materials synthesis
- HyperTune: Dynamic Hyperparameter Tuning For Efficient Distribution of DNN Training Over Heterogeneous Systems
- Deeplite Neutrino: An End-to-End Framework for Constrained Deep Learning Model Optimization
- A Scalable and Cloud-Native Hyperparameter Tuning System
- Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers
- Benchmark Tests of Convolutional Neural Network and Graph Convolutional Network on HorovodRunner Enabled Spark Clusters
- A Survey on Large-scale Machine Learning
- A Simple Non-i.i.d. Sampling Approach for Efficient Training and Better Generalization
- Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework
- Computational Causal Inference
- An Efficient DP-SGD Mechanism for Large Scale NLP Models
- Context-Aware Drive-thru Recommendation Service at Fast Food Restaurants
- Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients
- CSER: Communication-efficient SGD with Error Reset
- Integrating Deep Learning in Domain Sciences at Exascale
- On the Convergence of Quantized Parallel Restarted SGD for Central Server Free Distributed Training
- Parallelized Training of Restricted Boltzmann Machines using Markov-Chain Monte Carlo Methods
- GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent
- Training Multiscale-CNN for Large Microscopy Image Classification in One Hour
- CrossoverScheduler: Overlapping Multiple Distributed Training Applications in a Crossover Manner
- OD-SGD: One-step Delay Stochastic Gradient Descent for Distributed Training
- Elastic deep learning in multi-tenant GPU cluster
- HPC AI500: A Benchmark Suite for HPC AI Systems
- A Linear Algebraic Approach to Model Parallelism in Deep Learning
- Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension
- Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping
- Shuffle-Exchange Brings Faster: Reduce the Idle Time During Communication for Decentralized Neural Network Training
- Coded Elastic Computing
- ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
- BFTrainer: Low-Cost Training of Neural Networks on Unfillable Supercomputer Nodes
- A Predictive Autoscaler for Elastic Batch Jobs
- Towards automated neural design: An open source, distributed neural architecture research framework
- Neural Architecture Search in operational context: a remote sensing case-study
- Task allocation for decentralized training in heterogeneous environment
- Dynamic Gradient Aggregation for Federated Domain Adaptation
- Hoplite: Efficient and Fault-Tolerant Collective Communication for Task-Based Distributed Systems
- Cyclic orthogonal convolutions for long-range integration of features
- Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization
- Multilingual Translation via Grafting Pre-trained Language Models
- HADFL: Heterogeneity-aware Decentralized Federated Learning Framework
- Secure Distributed Training at Scale
- Logically Consistent Loss for Visual Question Answering
- Workload-aware Automatic Parallelization for Multi-GPU DNN Training
- MXDAG: A Hybrid Abstraction for Cluster Applications
- ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table
- HetSeq: Distributed GPU Training on Heterogeneous Infrastructure
- Attention based on-device streaming speech recognition with large speech corpus
- Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework
- Cascade RNN-Transducer: Syllable Based Streaming On-device Mandarin Speech Recognition with a Syllable-to-Character Converter
- Deploying Scientific AI Networks at Petaflop Scale on Secure Large Scale HPC Production Systems with Containers
- A Random Gossip BMUF Process for Neural Language Modeling
- MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training
- Oscars: Adaptive Semi-Synchronous Parallel Model for Distributed Deep Learning with Global View