212 citations · 457 across the 29 of their papers we have counts for
21 papers · 1 filter
Towards Robust Graph Incremental Learning on Evolving Graphs
Junwei Su, Difan Zou, Zijun Zhang +1
Incremental learning is a machine learning approach that involves training a model on a sequence of tasks, rather than all tasks at once. This ability to learn incrementally from a…
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
Yuan Cao, Difan Zou, Yuanzhi Li +1
We study the implicit bias of batch normalization trained by gradient descent. We show that when learning a linear model with batch normalization for binary classification, gradien…
The Benefits of Mixup for Feature Learning
Difan Zou, Yuan Cao, Yuanzhi Li +1
Mixup, a simple data augmentation method that randomly mixes two data points via linear interpolation, has been extensively applied in various deep learning applications to gain be…
Finite-Sample Analysis of Learning High-Dimensional Single ReLU Neuron
Jingfeng Wu, Difan Zou, Zixiang Chen +3
This paper considers the problem of learning a single ReLU neuron with squared loss (a.k.a., ReLU regression) in the overparameterized regime, where the input dimension can exceed…
The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift
Jingfeng Wu, Difan Zou, Vladimir Braverman +2
We study linear regression under covariate shift, where the marginal distribution over the input covariates differs in the source and the target domains, while the conditional dist…
Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regime
Difan Zou, Jingfeng Wu, Vladimir Braverman +2
Stochastic gradient descent (SGD) has achieved great success due to its superior performance in both optimization and generalization. Most of existing generalization analyses are m…
Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regression
Jingfeng Wu, Difan Zou, Vladimir Braverman +2
Stochastic gradient descent (SGD) has been shown to generalize well in many deep learning applications. In practice, one often runs SGD with a geometrically decaying stepsize, i.e.…
Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
Difan Zou, Yuan Cao, Yuanzhi Li +1
Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient desce…
The Benefits of Implicit Regularization from SGD in Least Squares Problems
Difan Zou, Jingfeng Wu, Vladimir Braverman +3
Stochastic gradient descent (SGD) exhibits strong algorithmic regularization effects in practice, which has been hypothesized to play an important role in the generalization of mod…
Self-training Converts Weak Learners to Strong Learners in Mixture Models
Spencer Frei, Difan Zou, Zixiang Chen +1
We consider a binary classification problem when the data comes from a mixture of two rotationally symmetric distributions satisfying concentration and anti-concentration propertie…
Provable Robustness of Adversarial Training for Learning Halfspaces with Noise
Difan Zou, Spencer Frei, Quanquan Gu
We analyze the properties of adversarial training for learning adversarially robust halfspaces in the presence of agnostic label noise. Denoting as the best ro…
Benign Overfitting of Constant-Stepsize SGD for Linear Regression
Difan Zou, Jingfeng Wu, Vladimir Braverman +2
There is an increasing realization that algorithmic inductive biases are central in preventing overfitting; empirically, we often see a benign overfitting phenomenon in overparamet…
Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate
Jingfeng Wu, Difan Zou, Vladimir Braverman +1
Understanding the algorithmic bias of \emph{stochastic gradient descent} (SGD) is one of the key challenges in modern machine learning and deep learning theory. Most of the existin…
Faster Convergence of Stochastic Gradient Langevin Dynamics for Non-Log-Concave Sampling
Difan Zou, Pan Xu, Quanquan Gu
We provide a new convergence analysis of stochastic gradient Langevin dynamics (SGLD) for sampling from a class of distributions that can be non-log-concave. At the core of our app…
On the Global Convergence of Training Deep Linear ResNets
Difan Zou, Philip M. Long, Quanquan Gu
We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training -hidden-layer linear residual networks (ResNets). We prove that for training…
Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks
Difan Zou, Ziniu Hu, Yewen Wang +3
Graph convolutional networks (GCNs) have recently received wide attentions, due to their successful applications in different graph tasks and different domains. Training GCNs for a…
Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo
Bao Wang, Difan Zou, Quanquan Gu +1
As an important Markov Chain Monte Carlo (MCMC) method, stochastic gradient Langevin dynamics (SGLD) algorithm has achieved great success in Bayesian learning and posterior samplin…
How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
Zixiang Chen, Yuan Cao, Difan Zou +1
A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the…
An Improved Analysis of Training Over-parameterized Deep Neural Networks
Difan Zou, Quanquan Gu
A recent line of research has shown that gradient-based algorithms with random initialization can converge to the global minima of the training loss for over-parameterized (i.e., s…
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
Difan Zou, Yuan Cao, Dongruo Zhou +1
We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, w…
Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently
Yaodong Yu, Difan Zou, Quanquan Gu
We propose a family of nonconvex optimization algorithms that are able to save gradient and negative curvature computations to a large extent, and are guaranteed to find an approxi…