Stochastic Optimization with Laggard Data Pipelines

Abstract

State-of-the-art optimization is steadily shifting towards massively parallel pipelines with extremely large batch sizes. As a consequence, CPU-bound preprocessing and disk/memory/network operations have emerged as new performance bottlenecks, as opposed to hardware-accelerated gradient computations. In this regime, a recently proposed approach is data echoing (Choi et al., 2019), which takes repeated gradient steps on the same batch while waiting for fresh data to arrive from upstream. We provide the first convergence analyses of "data-echoed" extensions of common optimization methods, showing that they exhibit provable improvements over their synchronous counterparts. Specifically, we show that in convex optimization with stochastic minibatches, data echoing affords speedups on the curvature-dominated part of the convergence rate, while maintaining the optimal statistical rate.

arXiv commentPublished as a conference paper at NeurIPS 2020

References in corpus (19)

• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Don't Decay the Learning Rate, Increase the Batch Size
• Large Batch Training of Convolutional Networks
• Train faster, generalize better: Stability of stochastic gradient descent
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Measuring the Effects of Data Parallelism on Neural Network Training
• An Empirical Model of Large-Batch Training
• Image Classification at Supercomputer Scale
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• Characterizing Deep-Learning I/O Workloads in TensorFlow
• Augment your batch: better training with larger batches
• Is Local SGD Better than Minibatch SGD?
• An Exponential Learning Rate Schedule for Deep Learning
• Stability and Convergence Trade-off of Iterative Optimization Algorithms
• Faster Neural Network Training with Data Echoing
• Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch-Prox
• Accelerating Reinforcement Learning through GPU Atari Emulation