Adaptive Braking for Mitigating Gradient Delay

Abstract

Neural network training is commonly accelerated by using multiple synchronized workers to compute gradient updates in parallel. Asynchronous methods remove synchronization overheads and improve hardware utilization at the cost of introducing gradient delay, which impedes optimization and can lead to lower final model performance. We introduce Adaptive Braking (AB), a modification for momentum-based optimizers that mitigates the effects of gradient delay. AB dynamically scales the gradient based on the alignment of the gradient and the velocity. This can dampen oscillations along high curvature directions of the loss surface, stabilizing and accelerating asynchronous training. We show that applying AB on top of SGD with momentum enables training ResNets on CIFAR-10 and ImageNet-1k with delays $D\ge$ 32 update steps with minimal drop in final test accuracy.

arXiv commentIn Beyond First Order Methods in ML Systems workshop at the 37th International Conference on Machine Learning, 2020

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Revisiting Distributed Synchronous SGD
• Staleness-aware Async-SGD for Distributed Deep Learning
• Asynchronous Stochastic Gradient Descent with Delay Compensation
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• PipeMare: Asynchronous Pipeline Parallel DNN Training
• Adaptive Restart for Accelerated Gradient Schemes
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• Gradient Energy Matching for Distributed Asynchronous Gradient Descent

Cited by in corpus (1)

• Pipelined Backpropagation at Scale: Training Large Models without Batches