86 citations · 120 across the 18 of their papers we have counts for
19 papers
Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks
Yuhang Cai, Kangjie Zhou, Jingfeng Wu +3
We establish the asymptotic implicit bias of gradient descent (GD) for generic non-homogeneous deep networks under exponential loss. Specifically, we characterize three key propert…
Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression
Jingfeng Wu, Peter Bartlett, Matus Telgarsky +1
In overparameterized logistic regression, gradient descent (GD) iterates diverge in norm while converging in direction to the maximum -margin solution -- a phenomenon known…
Scaling Laws in Linear Regression: Compute, Parameters, and Data
Licong Lin, Jingfeng Wu, Sham M. Kakade +2
Empirically, large-scale deep learning models often satisfy a neural scaling law: the test error of the trained model improves polynomially as the model size and data size grow. Ho…
Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency
Jingfeng Wu, Peter L. Bartlett, Matus Telgarsky +1
We consider gradient descent (GD) with a constant stepsize applied to logistic regression with linearly separable data, where the constant stepsize is so large that the loss in…
Risk Bounds of Accelerated SGD for Overparameterized Linear Regression
Xuheng Li, Yihe Deng, Jingfeng Wu +2
Accelerated stochastic gradient descent (ASGD) is a workhorse in deep learning and often achieves better generalization performance than SGD. However, existing optimization theory…
How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?
Jingfeng Wu, Difan Zou, Zixiang Chen +3
Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities, enabling them to solve unseen tasks solely based on input contexts without adjus…
Private Federated Frequency Estimation: Adapting to the Hardness of the Instance
Jingfeng Wu, Wennan Zhu, Peter Kairouz +1
In federated frequency estimation (FFE), multiple clients work together to estimate the frequencies of their collective data by communicating with a server that respects the privac…
Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability
Jingfeng Wu, Vladimir Braverman, Jason D. Lee
Recent research has observed that in machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) [Cohen, et al., 2021], where the stepsizes a…
Fixed Design Analysis of Regularization-Based Continual Learning
Haoran Li, Jingfeng Wu, Vladimir Braverman
We consider a continual learning (CL) problem with two linear regression tasks in the fixed design setting, where the feature vectors are assumed fixed and the labels are assumed t…
Finite-Sample Analysis of Learning High-Dimensional Single ReLU Neuron
Jingfeng Wu, Difan Zou, Zixiang Chen +3
This paper considers the problem of learning a single ReLU neuron with squared loss (a.k.a., ReLU regression) in the overparameterized regime, where the input dimension can exceed…
The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift
Jingfeng Wu, Difan Zou, Vladimir Braverman +2
We study linear regression under covariate shift, where the marginal distribution over the input covariates differs in the source and the target domains, while the conditional dist…
Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regime
Difan Zou, Jingfeng Wu, Vladimir Braverman +2
Stochastic gradient descent (SGD) has achieved great success due to its superior performance in both optimization and generalization. Most of existing generalization analyses are m…
Gap-Dependent Unsupervised Exploration for Reinforcement Learning
Jingfeng Wu, Vladimir Braverman, Lin F. Yang
For the problem of task-agnostic reinforcement learning (RL), an agent first collects samples from an unknown environment without the supervision of reward signals, then is reveale…
The Benefits of Implicit Regularization from SGD in Least Squares Problems
Difan Zou, Jingfeng Wu, Vladimir Braverman +3
Stochastic gradient descent (SGD) exhibits strong algorithmic regularization effects in practice, which has been hypothesized to play an important role in the generalization of mod…
Benign Overfitting of Constant-Stepsize SGD for Linear Regression
Difan Zou, Jingfeng Wu, Vladimir Braverman +2
There is an increasing realization that algorithmic inductive biases are central in preventing overfitting; empirically, we often see a benign overfitting phenomenon in overparamet…
Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learning
Jingfeng Wu, Vladimir Braverman, Lin F. Yang
In this paper we consider multi-objective reinforcement learning where the objectives are balanced using preferences. In practice, the preferences are often given in an adversarial…
Obtaining Adjustable Regularization for Free via Iterate Averaging
Jingfeng Wu, Vladimir Braverman, Lin F. Yang
Regularization for optimization is a crucial technique to avoid overfitting in machine learning. In order to obtain the best performance, we usually train a model by tuning the reg…
Tangent-Normal Adversarial Regularization for Semi-supervised Learning
Bing Yu, Jingfeng Wu, Jinwen Ma +1
Compared with standard supervised learning, the key difficulty in semi-supervised learning is how to make full use of the unlabeled data. A recently proposed method, virtual advers…
The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
Zhanxing Zhu, Jingfeng Wu, Bing Yu +2
Understanding the behavior of stochastic gradient descent (SGD) in the context of deep neural networks has raised lots of concerns recently. Along this line, we study a general for…