1 citations · 2 across the 10 of their papers we have counts for
11 papers
Large Stepsizes Federated Learning on Logistic Regression with Linearly Separable Data: The Case of Heterogeneous Devices
Hok Fong Wong, Hoi-To Wai, Chung-Yiu Yau
This paper revisits the distributed learning problem for training a multinomial logistic regression model with the Federated Averaging () algorithm. We concentrate…
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Zijian Zhang, Rizhen Hu, Athanasios Glentis +4
Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across tra…
EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization
Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3
Lookahead-based acceleration methods, such as Nesterov's momentum, are widely used in optimization, but they often become unreliable in deep learning training mainly due to stochas…
Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
Athanasios Glentis, Dawei Li, Chung-Yiu Yau +1
It is widely believed that stochastic gradient descent (SGD) performs significantly worse than adaptive optimizers such as Adam in pre-training Large Language Models (LLMs). Yet th…
RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models
Quan Wei, Chung-Yiu Yau, Hoi-To Wai +4
Supervised fine-tuning is a standard method for adapting pre-trained large language models (LLMs) to downstream tasks. Quantization has been recently studied as a post-training tec…
Decentralized Stochastic Optimization over Unreliable Networks via Two-timescales Updates
Haoming Liu, Chung-Yiu Yau, Hoi-To Wai
This paper introduces a robust two-timescale compressed primal-dual (TiCoPD) algorithm tailored for decentralized optimization under bandwidth-limited and unreliable channels. By i…
A Two-timescale Primal-dual Algorithm for Decentralized Optimization with Compression
Haoming Liu, Chung-Yiu Yau, Hoi-To Wai
This paper proposes a two-timescale compressed primal-dual (TiCoPD) algorithm for decentralized optimization with improved communication efficiency over prior works on primal-dual…
A Stochastic Approximation Approach for Efficient Decentralized Optimization on Random Networks
Chung-Yiu Yau, Haoming Liu, Hoi-To Wai
A challenging problem in decentralized optimization is to develop algorithms with fast convergence on random and time varying topologies under unreliable and bandwidth-constrained…
EMC: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence
Chung-Yiu Yau, Hoi-To Wai, Parameswaran Raman +2
A key challenge in contrastive learning is to generate negative samples from a large sample set to contrast with positive samples, for learning better encoding of the data. These n…
Multi-agent Performative Prediction with Greedy Deployment and Consensus Seeking Agents
Qiang Li, Chung-Yiu Yau, Hoi-To Wai
We consider a scenario where multiple agents are learning a common decision vector from data which can be influenced by the agents' decisions. This leads to the problem of multi-ag…
DoCoM: Compressed Decentralized Optimization with Near-Optimal Sample Complexity
Chung-Yiu Yau, Hoi-To Wai
This paper proposes the Doubly Compressed Momentum-assisted stochastic gradient tracking algorithm for communication-efficient decentralized optimization. The algo…