9 papers
Generalization Analysis of Biased Stochastic Gradient Methods for Minimax Problems
Shuang Zeng, Yunwen Lei, Yiming Ying
Minimax optimization refers to a class of optimization problems with some variables to minimize and other variables to maximize. Biased stochastic gradient methods have shown pract…
Approximation in Contrastive Representation Learning
Yuanfan Li, Zihan Zhang, Yiming Ying +1
Contrastive representation learning (CRL) is a fundamental paradigm for learning transferable representations. Recent work identifies its population target as the pointwise mutual…
Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
Yiming Ying
We study the identity straight-through estimator (STE) for training a two-layer binary-activation network with hinge loss from the perspective of Statistical Learning Theory (SLT).…
Minimax-Optimal Generalization Bounds for Smooth Deep Neural Networks Trained by (Stochastic) Gradient Descent
Junyu Zhou, Puyu Wang, Dennis Wagner +3
Characterizing the optimization dynamics and statistical performance of over-parameterized deep neural networks (DNNs) remains a central challenge in understanding the remarkable s…
Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks
Junyu Zhou, Puyu Wang, Yunwen Lei +2
Recent progress has been made in understanding the statistical generalization performance of gradient descent methods for overparameterized neural networks within the neural tangen…
Statistical Consistency and Generalization of Contrastive Representation Learning
Yuanfan Li, Xiyuan Wei, Tianbao Yang +1
Contrastive representation learning (CRL) underpins many modern foundation models. Despite recent theoretical progress, existing analyses suffer from several key limitations: (i) t…
Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification
Yuanfan Li, Yunwen Lei, Zheng-Chu Guo +1
Recent advances have significantly improved our understanding of the generalization performance of gradient descent (GD) methods in deep neural networks. A natural and fundamental…
How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Jun Chen, Hong Chen, Yonghua Yu +1
In recent years, contrastive learning has achieved state-of-the-art performance in the territory of self-supervised representation learning. Many previous works have attempted to p…
Differentially Private Non-convex Learning for Multi-layer Neural Networks
Hanpu Shen, Cheng-Long Wang, Zihang Xiang +2
This paper focuses on the problem of Differentially Private Stochastic Optimization for (multi-layer) fully connected neural networks with a single output node. In the first part,…