37 citations · 43 across the 5 of their papers we have counts for
5 papers · 1 filter
Memory-Statistics Tradeoff in Continual Learning with Structural Regularization
Haoran Li, Jingfeng Wu, Vladimir Braverman
We study the statistical performance of a continual learning problem with two linear regression tasks in a well-specified random design setting. We consider a structural regulariza…
Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regression
Jingfeng Wu, Difan Zou, Vladimir Braverman +2
Stochastic gradient descent (SGD) has been shown to generalize well in many deep learning applications. In practice, one often runs SGD with a geometrically decaying stepsize, i.e.…
Lifelong Learning with Sketched Structural Regularization
Haoran Li, Aditya Krishnan, Jingfeng Wu +3
Preventing catastrophic forgetting while continually learning new tasks is an essential problem in lifelong learning. Structural regularization (SR) refers to a family of algorithm…
Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate
Jingfeng Wu, Difan Zou, Vladimir Braverman +1
Understanding the algorithmic bias of \emph{stochastic gradient descent} (SGD) is one of the key challenges in modern machine learning and deep learning theory. Most of the existin…
On the Noisy Gradient Descent that Generalizes as SGD
Jingfeng Wu, Wenqing Hu, Haoyi Xiong +3
The gradient noise of SGD is considered to play a central role in the observed strong generalization abilities of deep learning. While past studies confirm that the magnitude and t…