32 citations · 51 across the 11 of their papers we have counts for
12 papers
Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature
Zihan Zhu, Zhehang Du, Xuyang Chen +5
Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapt…
Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers
Tim Tsz-Kit Lau, Weijie Su
A striking geometric disparity has long persisted in the practice of deep learning. While modern neural network architectures naturally exhibit rich symmetry and equivariance prope…
PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
Tim Tsz-Kit Lau, Qi Long, Weijie Su
The ever-growing scale of deep learning models and training data underscores the critical importance of efficient optimization methods. While preconditioned gradient methods such a…
Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism
Tim Tsz-Kit Lau, Weijian Li, Chenwei Xu +2
An appropriate choice of batch sizes in large-scale model training is crucial, yet it involves an intrinsic yet inevitable dilemma: large-batch training improves training efficienc…
Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods
Tim Tsz-Kit Lau, Weijian Li, Chenwei Xu +2
Modern deep neural networks often require distributed training with many workers due to their large size. As the number of workers increases, communication overheads become the mai…
AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
Tim Tsz-Kit Lau, Han Liu, Mladen Kolar
The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-scale model training. Although larg…
Non-Log-Concave and Nonsmooth Sampling via Langevin Monte Carlo Algorithms
Tim Tsz-Kit Lau, Han Liu, Thomas Pock
We study the problem of approximate sampling from non-log-concave distributions, e.g., Gaussian mixtures, which is often challenging even in low dimensions due to their multimodali…
Bregman Proximal Langevin Monte Carlo via Bregman--Moreau Envelopes
Tim Tsz-Kit Lau, Han Liu
We propose efficient Langevin Monte Carlo algorithms for sampling distributions with nonsmooth convex composite potentials, which is the sum of a continuously differentiable functi…
The Multi-Agent Pickup and Delivery Problem: MAPF, MARL and Its Warehouse Applications
Tim Tsz-Kit Lau, Biswa Sengupta
We study two state-of-the-art solutions to the multi-agent pickup and delivery (MAPD) problem based on different principles -- multi-agent path-finding (MAPF) and multi-agent reinf…
Wasserstein Distributionally Robust Optimization with Wasserstein Barycenters
Tim Tsz-Kit Lau, Han Liu
In many applications in statistics and machine learning, the availability of data samples from multiple possibly heterogeneous sources has become increasingly prevalent. On the oth…
A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training
Tim Tsz-Kit Lau, Jinshan Zeng, Baoyuan Wu +1
Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. The backpropagation (backprop) algorithm has long been the most…
Global Convergence of Block Coordinate Descent in Deep Learning
Jinshan Zeng, Tim Tsz-Kit Lau, Shaobo Lin +1
Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in dee…