1 citations · 3 across the 12 of their papers we have counts for
16 papers
Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference
Tianxiang Xia, Kai Lion, Niao He +3
Scaling laws continue to reward increases in model scale, while the resulting computational and memory demands drive greater reliance on cloud infrastructure for inference. Yet unr…
Rethinking Routing Exploration in Modern Mixture-of-Experts
Bingcong Li, Hao Ma, Michael Muehlebach +2
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained exper…
Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics
Yudong Wei, Liang Zhang, Bingcong Li +1
Low-rank matrix optimization is often carried out via the Burer-Monteiro (BM) formulation, but choosing the factorization rank is delicate and can substantially slow optimizati…
Integrated electro-optic attention nonlinearities for transformers
Luis Mickeler, Kai Lion, Alfonso Nardi +5
Transformers achieve state-of-the-art performance in language processing and computer vision. Their computation increasingly depends on hardware that can keep pace not only with la…
Zeroth-Order Optimization at the Edge of Stability
Minhak Song, Liang Zhang, Bingcong Li +3
Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and memory-efficient fine-tuning of large models,…
SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inference
Hao Ma, Melis Ilayda Bal, Liang Zhang +4
Modern large language models are increasingly deployed under compute and memory constraints, making flexible control of model capacity a central challenge. While sparse and low-ran…
On the Benefits of Weight Normalization for Overparameterized Matrix Sensing
Yudong Wei, Liang Zhang, Bingcong Li +1
While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized…
Zeroth-Order Optimization Finds Flat Minima
Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil +3
Zeroth-order methods are extensively used in machine learning applications where gradients are infeasible or expensive to compute, such as black-box attacks, reinforcement learning…
PoLAR: Polar-Decomposed Low-Rank Adapter Representation
Kai Lion, Liang Zhang, Bingcong Li +1
We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performanc…
Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems
Bingcong Li, Liang Zhang, Niao He
Sharpness-aware minimization (SAM) improves generalization of various deep learning tasks. Motivated by popular architectures such as LoRA, we explore the implicit regularization o…
On the Crucial Role of Initialization for Matrix Factorization
Bingcong Li, Liang Zhang, Aryan Mokhtari +1
This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth…
A Hessian-Aware Stochastic Differential Equation for Modelling SGD
Xiang Li, Zebang Shen, Liang Zhang +1
Continuous-time approximation of Stochastic Gradient Descent (SGD) is a crucial tool to study its escaping behaviors from stationary points. However, existing stochastic differenti…
Primal Methods for Variational Inequality Problems with Functional Constraints
Liang Zhang, Niao He, Michael Muehlebach
Variational inequality problems are recognized for their broad applications across various fields including machine learning and operations research. First-order methods have emerg…
Parameter-Agnostic Optimization under Relaxed Smoothness
Florian Hübler, Junchi Yang, Xiang Li +1
Tuning hyperparameters, such as the stepsize, presents a major challenge of training machine learning models. To address this challenge, numerous adaptive optimization algorithms h…
Optimal Guarantees for Algorithmic Reproducibility and Gradient Complexity in Convex Optimization
Liang Zhang, Junchi Yang, Amin Karbasi +1
Algorithmic reproducibility measures the deviation in outputs of machine learning algorithms upon minor changes in the training process. Previous work suggests that first-order met…
DPZero: Private Fine-Tuning of Language Models without Backpropagation
Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil +2
The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy. First, as the size of LLMs continues t…