76 citations · 110 across the 10 of their papers we have counts for
4 papers · 2 filters
Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Zijian Zhang, Rizhen Hu, Athanasios Glentis +4
Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across tra…
A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
Tian Ding, Dawei Li, Ruoyu Sun
We investigate the geometric structure of stationary plateaus that arise in the loss landscape of two-layer neural networks with smooth activation functions. We focus on the phenom…
EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization
Chung-Yiu Yau, Dawei Li, Athanasios Glentis +3
Lookahead-based acceleration methods, such as Nesterov's momentum, are widely used in optimization, but they often become unreliable in deep learning training mainly due to stochas…
Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates
Athanasios Glentis, Dawei Li, Chung-Yiu Yau +1
It is widely believed that stochastic gradient descent (SGD) performs significantly worse than adaptive optimizers such as Adam in pre-training Large Language Models (LLMs). Yet th…