How Data Augmentation affects Optimization for Linear Regression

Abstract

Though data augmentation has rapidly emerged as a key tool for optimization in modern machine learning, a clear picture of how augmentation schedules affect optimization and interact with optimization hyperparameters such as learning rate is nascent. In the spirit of classical convex optimization and recent work on implicit bias, the present work analyzes the effect of augmentation on optimization in the simple convex setting of linear regression with MSE loss. We find joint schedules for learning rate and data augmentation scheme under which augmented gradient descent provably converges and characterize the resulting minimum. Our results apply to arbitrary augmentation schemes, revealing complex interactions between learning rates and augmentations even in the convex setting. Our approach interprets augmented (S)GD as a stochastic optimization method for a time-varying sequence of proxy losses. This gives a unified way to analyze learning rate, batch size, and augmentations ranging from additive noise to random projections. From this perspective, our results, which also give rates of convergence, can be viewed as Monro-Robbins type conditions for augmented (S)GD.

arXiv comment31 pages, 3 figures, NeurIPS 2021

References in corpus (14)

• mixup: Beyond Empirical Risk Minimization
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Integration with respect to the Haar measure on unitary, orthogonal and symplectic group
• Don't Decay the Learning Rate, Increase the Batch Size
• Benign Overfitting in Linear Regression
• Dropout Training as Adaptive Regularization
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• A Kernel Theory of Modern Data Augmentation
• Implicit Rugosity Regularization via Data Augmentation
• The Penalty Imposed by Ablated Data Augmentation