8 papers
Group Distributionally Robust Optimization over Decentralized Networks
Haomin Bai, Junyan Sun, Quan Cheng +2
We study group distributionally robust optimization (GDRO) over decentralized networks, where workers collaborate to learn robust models by minimizing the maximum local risk. Exist…
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Sifan Yang, Hongyi Tao, Dingzhi Yu +5
Memory-efficient subspace optimization has emerged as a practical approach to reducing the memory cost of LLM training while preserving optimization performance. Previous algorithm…
Last-Iterate Convergence of Clipped SGD with Linear Decay
Wei Jiang, Haomin Bai, Zechao Li
In this paper, we study the last-iterate convergence of clipped stochastic gradient descent under heavy-tailed noise. We assume that the stochastic gradients are unbiased and that…
Dimension-Free Decentralized Nonsmooth Nonconvex Stochastic Optimization
Yuanyu Wan, Lan Xue, Haomin Bai +2
We investigate decentralized nonsmooth nonconvex stochastic optimization over a network of nodes, with the goal of finding an -Goldstein stationary point. The best exist…
Communication-Efficient Agnostic Federated Learning via Faster Convergence and Compression
Haomin Bai, Junyan Sun, Sifan Yang +2
Agnostic federated learning (AFL) seeks a model that performs reliably across heterogeneous workers, but communication remains a bottleneck. We improve communication efficiency…
Group Distributionally Robust Optimization with Flexible Sample Queries
Haomin Bai, Dingzhi Yu, Shuai Li +2
Group distributionally robust optimization (GDRO) aims to develop models that perform well across distributions simultaneously. Existing GDRO algorithms can only process a fixe…
Efficient Stochastic Approximation of Minimax Excess Risk Optimization
Lijun Zhang, Haomin Bai, Wei-Wei Tu +2
While traditional distributionally robust optimization (DRO) aims to minimize the maximal risk over a set of distributions, Agarwal and Zhang (2022) recently proposed a variant tha…
Stochastic Approximation Approaches to Group Distributionally Robust Optimization and Beyond
Lijun Zhang, Haomin Bai, Peng Zhao +2
This paper investigates group distributionally robust optimization (GDRO) with the goal of learning a model that performs well over different distributions. First, we formulate…