30 citations · 148 across the 30 of their papers we have counts for
25 papers · 1 filter
Stochastic Gradient Descent with Momentum is Algorithmically Stable
Yunwen Lei, Zimeng Wang, Xiaoming Yuan
Stochastic gradient descent with momentum (SGDM) is one of the most widely used optimization algorithms in machine learning. While optimization properties of SGDM have been extensi…
Learning Theory of the SVRG: Generalization and Convergence Analysis
Yunwen Lei, Zimeng Wang, Xiaoming Yuan
Variance reduction (VR) methods employ stochastic gradients with decreasing variance, and they have been widely applied to solve large-scale optimization problems in machine learni…
Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networks
Yunwen Lei, Yufeng Xie
Overparameterized neural networks often show a benign overfitting property in the sense of achieving excellent generalization behavior despite the number of parameters exceeding th…
Generalization Guarantees of Gradient Descent for Multi-Layer Neural Networks
Puyu Wang, Yunwen Lei, Di Wang +2
Recently, significant progress has been made in understanding the generalization of neural networks (NNs) trained by gradient descent (GD) using the algorithmic stability approach.…
Generalization Analysis for Contrastive Representation Learning
Yunwen Lei, Tianbao Yang, Yiming Ying +1
Recently, contrastive learning has found impressive success in advancing the state of the art in solving various machine learning tasks. However, the existing generalization analys…
Generalization Bounds for Inductive Matrix Completion in Low-noise Settings
Antoine Ledent, Rodrigo Alves, Yunwen Lei +2
We study inductive matrix completion (matrix completion with side information) under an i.i.d. subgaussian noise assumption at a low noise regime, with uniform sampling of the entr…
On Stability and Generalization of Bilevel Optimization Problem
Meng Ding, Mingxi Lei, Yunwen Lei +2
(Stochastic) bilevel optimization is a frequently encountered problem in machine learning with a wide range of applications such as meta-learning, hyper-parameter optimization, and…
Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks
Yunwen Lei, Rong Jin, Yiming Ying
While significant theoretical progress has been achieved, unveiling the generalization mystery of overparameterized neural networks still remains largely elusive. In this paper, we…
Differentially Private SGDA for Minimax Problems
Zhenhuan Yang, Shu Hu, Yunwen Lei +3
Stochastic gradient descent ascent (SGDA) and its variants have been the workhorse for solving minimax problems. However, in contrast to the well-studied stochastic gradient descen…
Simple Stochastic and Online Gradient Descent Algorithms for Pairwise Learning
Zhenhuan Yang, Yunwen Lei, Puyu Wang +2
Pairwise learning refers to learning tasks where the loss function depends on a pair of instances. It instantiates many important machine learning tasks such as bipartite ranking a…
Stability and Generalization for Randomized Coordinate Descent
Puyu Wang, Liang Wu, Yunwen Lei
Randomized coordinate descent (RCD) is a popular optimization algorithm with wide applications in solving various machine learning problems, which motivates a lot of theoretical an…
Fine-grained Generalization Analysis of Structured Output Prediction
Waleed Mustafa, Yunwen Lei, Antoine Ledent +1
In machine learning we often encounter structured output prediction problems (SOPPs), i.e. problems where the output space admits a rich internal structure. Application domains whe…
Stability and Generalization of Stochastic Gradient Methods for Minimax Problems
Yunwen Lei, Zhenhuan Yang, Tianbao Yang +1
Many machine learning problems can be formulated as minimax problems such as Generative Adversarial Networks (GANs), AUC maximization and robust estimation, to mention but a few. A…
Fine-grained Generalization Analysis of Vector-valued Learning
Liang Wu, Antoine Ledent, Yunwen Lei +1
Many fundamental machine learning tasks can be formulated as a problem of learning with vector-valued functions, where we learn multiple scalar-valued functions together. Although…
Stochastic Hard Thresholding Algorithms for AUC Maximization
Zhenhuan Yang, Baojian Zhou, Yunwen Lei +1
In this paper, we aim to develop stochastic hard thresholding algorithms for the important problem of AUC maximization in imbalanced classification. The main challenge is the pairw…
Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent
Yunwen Lei, Yiming Ying
Recently there are a considerable amount of work devoted to the study of the algorithmic stability and generalization for stochastic gradient descent (SGD). However, the existing s…
On Performance Estimation in Automatic Algorithm Configuration
Shengcai Liu, Ke Tang, Yunwen Lei +1
Over the last decade, research on automated parameter tuning, often referred to as automatic algorithm configuration (AAC), has made significant progress. Although the usefulness o…
Stochastic Proximal AUC Maximization
Yunwen Lei, Yiming Ying
In this paper we consider the problem of maximizing the Area under the ROC curve (AUC) which is a widely used performance metric in imbalanced classification and anomaly detection.…
Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions
Yunwen Lei, Ting Hu, Guiying Li +1
Stochastic gradient descent (SGD) is a popular and efficient method with wide applications in training deep neural nets and other nonconvex models. While the behavior of SGD is wel…
Convergence of Online Mirror Descent
Yunwen Lei, Ding-Xuan Zhou
In this paper we consider online mirror descent (OMD) algorithms, a class of scalable online learning algorithms exploiting data geometric structures through mirror maps. Necessary…
Convergence of Unregularized Online Learning Algorithms
Yunwen Lei, Lei Shi, Zheng-Chu Guo
In this paper we study the convergence of online gradient descent algorithms in reproducing kernel Hilbert spaces (RKHSs) without regularization. We establish a sufficient conditio…
Data-dependent Generalization Bounds for Multi-class Classification
Yunwen Lei, Urun Dogan, Ding-Xuan Zhou +1
In this paper, we study data-dependent generalization error bounds exhibiting a mild dependency on the number of classes, making them suitable for multi-class learning with a large…
Local Rademacher Complexity-based Learning Guarantees for Multi-Task Learning
Niloofar Yousefi, Yunwen Lei, Marius Kloft +2
We show a Talagrand-type concentration inequality for Multi-Task Learning (MTL), using which we establish sharp excess risk bounds for MTL in terms of distribution- and data-depend…
Multi-class SVMs: From Tighter Data-Dependent Generalization Bounds to Novel Algorithms
Yunwen Lei, Ürün Dogan, Alexander Binder +1
This paper studies the generalization performance of multi-class classification algorithms, for which we obtain, for the first time, a data-dependent generalization error bound wit…