170 citations · 909 across the 66 of their papers we have counts for
9 papers · 1 filter
Robust Implicit Regularization via Weight Normalization
Hung-Hsu Chou, Holger Rauhut, Rachel Ward
Overparameterized models may have many interpolating solutions; implicit regularization refers to the hidden preference of a particular optimization method towards a certain interp…
On the fast convergence of minibatch heavy ball momentum
Raghu Bollapragada, Tyler Chen, Rachel Ward
Simple stochastic momentum methods are widely used in machine learning optimization, but their good practical performance is at odds with an absence of theoretical guarantees of ac…
How catastrophic can catastrophic forgetting be in linear regression?
Itay Evron, Edward Moroshko, Rachel Ward +2
To better understand catastrophic forgetting, we study fitting an overparameterized linear model to a sequence of tasks with different input distributions. We analyze how much the…
Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Space
Juncai He, Richard Tsai, Rachel Ward
The low-dimensional manifold hypothesis posits that the data found in many applications, such as those involving natural images, lie (approximately) on low-dimensional manifolds em…
Sample Efficiency of Data Augmentation Consistency Regularization
Shuo Yang, Yijun Dong, Rachel Ward +3
Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to…
SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning
Yuege Xie, Bobby Shi, Hayden Schaeffer +1
Sparse shrunk additive models and sparse random feature models have been developed separately as methods to learn low-order functions, where there are few interactions between vari…
Overparameterization and generalization error: weighted trigonometric interpolation
Yuege Xie, Hung-Hsu Chou, Holger Rauhut +1
Motivated by surprisingly good generalization properties of learned deep neural networks in overparameterized scenarios and by the related double descent phenomenon, this paper ana…
Implicit Regularization and Convergence for Weight Normalization
Xiaoxia Wu, Edgar Dobriban, Tongzheng Ren +5
Normalization methods such as batch [Ioffe and Szegedy, 2015], weight [Salimansand Kingma, 2016], instance [Ulyanov et al., 2016], and layer normalization [Baet al., 2016] have bee…
Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
Xiaoxia Wu, Simon S. Du, Rachel Ward
Adaptive gradient methods like AdaGrad are widely used in optimizing neural networks. Yet, existing convergence guarantees for adaptive gradient methods require either convexity or…