3 citations · 9 across the 13 of their papers we have counts for
15 papers
Lower Bounds for Anytime Acceleration of Gradient Descent
Chung-En Tsai, Ilyas Fatkhullin, Liang Zhang +1
Recent work suggests that the convergence rate of gradient descent (GD) in smooth convex optimization can be significantly improved by employing large stepsizes that may violate th…
Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics
Yudong Wei, Liang Zhang, Bingcong Li +1
Low-rank matrix optimization is often carried out via the Burer-Monteiro (BM) formulation, but choosing the factorization rank is delicate and can substantially slow optimizati…
Zeroth-Order Optimization at the Edge of Stability
Minhak Song, Liang Zhang, Bingcong Li +3
Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and memory-efficient fine-tuning of large models,…
Zeroth-Order Optimization Finds Flat Minima
Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil +3
Zeroth-order methods are extensively used in machine learning applications where gradients are infeasible or expensive to compute, such as black-box attacks, reinforcement learning…
PoLAR: Polar-Decomposed Low-Rank Adapter Representation
Kai Lion, Liang Zhang, Bingcong Li +1
We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performanc…
Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems
Bingcong Li, Liang Zhang, Niao He
Sharpness-aware minimization (SAM) improves generalization of various deep learning tasks. Motivated by popular architectures such as LoRA, we explore the implicit regularization o…
On the Crucial Role of Initialization for Matrix Factorization
Bingcong Li, Liang Zhang, Aryan Mokhtari +1
This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth…
A Hessian-Aware Stochastic Differential Equation for Modelling SGD
Xiang Li, Zebang Shen, Liang Zhang +1
Continuous-time approximation of Stochastic Gradient Descent (SGD) is a crucial tool to study its escaping behaviors from stationary points. However, existing stochastic differenti…
Primal Methods for Variational Inequality Problems with Functional Constraints
Liang Zhang, Niao He, Michael Muehlebach
Variational inequality problems are recognized for their broad applications across various fields including machine learning and operations research. First-order methods have emerg…
Optimal Guarantees for Algorithmic Reproducibility and Gradient Complexity in Convex Optimization
Liang Zhang, Junchi Yang, Amin Karbasi +1
Algorithmic reproducibility measures the deviation in outputs of machine learning algorithms upon minor changes in the training process. Previous work suggests that first-order met…
DPZero: Private Fine-Tuning of Language Models without Backpropagation
Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil +2
The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy. First, as the size of LLMs continues t…
Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization
Liang Zhang, Kiran Koshy Thekumparampil, Sewoong Oh +1
We study differentially private (DP) algorithms for smooth stochastic minimax optimization, with stochastic minimization as a byproduct. The holy grail of these settings is to guar…
Generalization Bounds of Nonconvex-(Strongly)-Concave Stochastic Minimax Optimization
Siqi Zhang, Yifan Hu, Liang Zhang +1
This paper takes an initial step to systematically investigate the generalization bounds of algorithms for solving nonconvex-(strongly)-concave (NC-SC/NC-C) stochastic minimax opti…
GCN-LASE: Towards Adequately Incorporating Link Attributes in Graph Convolutional Networks
Ziyao Li, Liang Zhang, Guojie Song
Graph Convolutional Networks (GCNs) have proved to be a most powerful architecture in aggregating local neighborhood information for individual graph nodes. Low-rank proximities an…
SepNE: Bringing Separability to Network Embedding
Ziyao Li, Liang Zhang, Guojie Song
Many successful methods have been proposed for learning low dimensional representations on large-scale networks, while almost all existing methods are designed in inseparable proce…