Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching

Abstract

Current techniques and systems for distributed model training mostly assume that clusters are comprised of homogeneous servers with a constant resource availability. However, cluster heterogeneity is pervasive in computing infrastructure, and is a fundamental characteristic of low-cost transient resources (such as EC2 spot instances). In this paper, we develop a dynamic batching technique for distributed data-parallel training that adjusts the mini-batch sizes on each worker based on its resource availability and throughput. Our mini-batch controller seeks to equalize iteration times on all workers, and facilitates training on clusters comprised of servers with different amounts of CPU and GPU resources. This variable mini-batch technique uses proportional control and ideas from PID controllers to find stable mini-batch sizes. Our empirical evaluation shows that dynamic batching can reduce model training times by more than 4x on heterogeneous clusters.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Hop: Heterogeneity-Aware Decentralized Training
• On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization
• Anytime MiniBatch: Exploiting Stragglers in Online Distributed Optimization
• Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers
• Speeding up Deep Learning with Transient Servers

Cited by in corpus (3)

• DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency for Federated Learning with Static and Streaming Dataset
• ScaDLES: Scalable Deep Learning over Streaming data at the Edge
• Accelerating Distributed ML Training via Selective Synchronization