212 citations · 799 across the 27 of their papers we have counts for
23 papers · 1 filter
Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization
Xuheng Li, Qiwei Di, Yuan Cao +1
The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this t…
Leveraging Inference-Time Compute for Diffusion Models via Global Scheduling of Denoising Trajectories
Yuan Cao, Yifu Tang, Hangqi Li +1
Diffusion models generate a sample by traversing a denoising trajectory, a sequence of stochastic noise-reduction steps that transforms pure noise into a draw from a target distrib…
Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
Yuanzhe Tao, Yifeng Liu, Huizhuo Yuan +3
Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process.…
Benign Overfitting in Two-Layer ReLU Convolutional Neural Networks for XOR Data
Xuran Meng, Difan Zou, Yuan Cao
Modern deep learning models are usually highly over-parameterized so that they can overfit the training data. Surprisingly, such overfitting neural networks can usually still achie…
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
Yuan Cao, Difan Zou, Yuanzhi Li +1
We study the implicit bias of batch normalization trained by gradient descent. We show that when learning a linear model with batch normalization for binary classification, gradien…
The Benefits of Mixup for Feature Learning
Difan Zou, Yuan Cao, Yuanzhi Li +1
Mixup, a simple data augmentation method that randomly mixes two data points via linear interpolation, has been extensively applied in various deep learning applications to gain be…
Benign Overfitting in Two-layer Convolutional Neural Networks
Yuan Cao, Zixiang Chen, Mikhail Belkin +1
Modern neural networks often have great expressive power and can be trained to overfit the training data, while still achieving a good test performance. This phenomenon is referred…
Benign Overfitting in Adversarially Robust Linear Classification
Jinghui Chen, Yuan Cao, Quanquan Gu
"Benign overfitting", where classifiers memorize noisy training data yet still achieve a good generalization performance, has drawn great attention in the machine learning communit…
Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
Difan Zou, Yuan Cao, Yuanzhi Li +1
Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient desce…
Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures
Yuan Cao, Quanquan Gu, Mikhail Belkin
Modern machine learning systems such as deep neural networks are often highly over-parameterized so that they can fit the noisy training data exactly, yet they can still achieve sm…
Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise
Spencer Frei, Yuan Cao, Quanquan Gu
We consider a one-hidden-layer leaky ReLU network of arbitrary width trained by stochastic gradient descent (SGD) following an arbitrary initialization. We prove that SGD produces…
Agnostic Learning of Halfspaces with Gradient Descent via Soft Margins
Spencer Frei, Yuan Cao, Quanquan Gu
We analyze the properties of gradient descent on convex surrogates for the zero-one loss for the agnostic learning of linear halfspaces. If is the best classificatio…
Agnostic Learning of a Single Neuron with Gradient Descent
Spencer Frei, Yuan Cao, Quanquan Gu
We consider the problem of learning the best-fitting single neuron as measured by the expected square loss over some unknown…
A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
Zixiang Chen, Yuan Cao, Quanquan Gu +1
A recent breakthrough in deep learning theory shows that the training of over-parameterized deep neural networks can be characterized by a kernel function called \textit{neural tan…
Towards Understanding the Spectral Bias of Deep Learning
Yuan Cao, Zhiying Fang, Yue Wu +2
An intriguing phenomenon observed during training neural networks is the spectral bias, which states that neural networks are biased towards learning less complex functions. The pr…
Tight Sample Complexity of Learning One-hidden-layer Convolutional Neural Networks
Yuan Cao, Quanquan Gu
We study the sample complexity of learning one-hidden-layer convolutional neural networks (CNNs) with non-overlapping filters. We propose a novel algorithm called approximate gradi…
How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
Zixiang Chen, Yuan Cao, Difan Zou +1
A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the…
Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networks
Spencer Frei, Yuan Cao, Quanquan Gu
The skip-connections used in residual networks have become a standard architecture choice in deep learning due to the increased training stability and generalization performance wi…
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
Yuan Cao, Quanquan Gu
We study the training and generalization of deep neural networks (DNNs) in the over-parameterized regime, where the network width (i.e., number of hidden nodes per layer) is much l…
Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
Yuan Cao, Quanquan Gu
Empirical studies show that gradient-based methods can learn deep neural networks (DNNs) with very good generalization performance in the over-parameterization regime, where DNNs c…
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
Difan Zou, Yuan Cao, Dongruo Zhou +1
We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, w…
On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
Dongruo Zhou, Jinghui Chen, Yuan Cao +2
Adaptive gradient methods are workhorses in deep learning. However, the convergence guarantees of adaptive gradient methods for nonconvex optimization have not been thoroughly stud…
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
Jinghui Chen, Dongruo Zhou, Yiqi Tang +3
Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed…