Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources

Abstract

With an increasing demand for training powers for deep learning algorithms and the rapid growth of computation resources in data centers, it is desirable to dynamically schedule different distributed deep learning tasks to maximize resource utilization and reduce cost. In this process, different tasks may receive varying numbers of machines at different time, a setting we call elastic distributed training. Despite the recent successes in large mini-batch distributed training, these methods are rarely tested in elastic distributed training environments and suffer degraded performance in our experiments, when we adjust the learning rate linearly immediately with respect to the batch size. One difficulty we observe is that the noise in the stochastic momentum estimation is accumulated over time and will have delayed effects when the batch size changes. We therefore propose to smoothly adjust the learning rate over time to alleviate the influence of the noisy momentum estimation. Our experiments on image classification, object detection and semantic segmentation have demonstrated that our proposed Dynamic SGD method achieves stabilized performance when varying the number of GPUs from 8 to 128. We also provide theoretical understanding on the optimality of linear learning rate scheduling and the effects of stochastic momentum.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Revisiting Distributed Synchronous SGD
• Don't Decay the Learning Rate, Increase the Batch Size
• Rethinking the Inception Architecture for Computer Vision
• Large Batch Training of Convolutional Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
• Three Factors Influencing Minima in SGD
• Asynchronous Stochastic Gradient Descent with Delay Compensation
• Bag of Freebies for Training Object Detection Neural Networks
• SlowFast Networks for Video Recognition
• Bag of Tricks for Image Classification with Convolutional Neural Networks
• An Empirical Model of Large-Batch Training
• AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• TVQA: Localized, Compositional Video Question Answering
• FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation

Cited by in corpus (5)

• ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation
• Hybrid Composition with IdleBlock: More Efficient Networks for Image Recognition
• Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning
• An Optimal Resource Allocator of Elastic Training for Deep Learning Jobs on Cloud
• Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters