1 citations · 2 across the 10 of their papers we have counts for
7 papers · 1 filter
Large Stepsizes Federated Learning on Logistic Regression with Linearly Separable Data: The Case of Heterogeneous Devices
Hok Fong Wong, Hoi-To Wai, Chung-Yiu Yau
This paper revisits the distributed learning problem for training a multinomial logistic regression model with the Federated Averaging () algorithm. We concentrate…
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Zijian Zhang, Rizhen Hu, Athanasios Glentis +4
Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across tra…
EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization
Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3
Lookahead-based acceleration methods, such as Nesterov's momentum, are widely used in optimization, but they often become unreliable in deep learning training mainly due to stochas…
Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
Athanasios Glentis, Dawei Li, Chung-Yiu Yau +1
It is widely believed that stochastic gradient descent (SGD) performs significantly worse than adaptive optimizers such as Adam in pre-training Large Language Models (LLMs). Yet th…
RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models
Quan Wei, Chung-Yiu Yau, Hoi-To Wai +4
Supervised fine-tuning is a standard method for adapting pre-trained large language models (LLMs) to downstream tasks. Quantization has been recently studied as a post-training tec…
EMC: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence
Chung-Yiu Yau, Hoi-To Wai, Parameswaran Raman +2
A key challenge in contrastive learning is to generate negative samples from a large sample set to contrast with positive samples, for learning better encoding of the data. These n…
DoCoM: Compressed Decentralized Optimization with Near-Optimal Sample Complexity
Chung-Yiu Yau, Hoi-To Wai
This paper proposes the Doubly Compressed Momentum-assisted stochastic gradient tracking algorithm for communication-efficient decentralized optimization. The algo…