4 papers · 1 filter
Trust Region Policy Distillation
Zhengpeng Xie, Li Lyna Zhang, Zeke Xie +1
Big goals are hard to achieve all at once; breaking them into small steps is wiser. We present Trust Region Policy Distillation (TOP-D), which transforms the notoriously unstable,…
Dimension-Free Saddle-Point Escape in Muon
Yanlin Long, Yufei Gu, Zeke Xie
Modern Large Language Model (LLM) training is fundamentally bottlenecked by pathologically flat saddle points in extreme high-dimensional landscapes. Motivated by this challenge, w…
MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
Yulong Huang, Xiang Liu, Hongxiang Huang +5
Linear Attention (LA) offers a promising paradigm for scaling large language models (LLMs) to long sequences by avoiding the quadratic complexity of self-attention. Recent LA model…
Mano: Restriking Manifold Optimization for LLM Training
Yufei Gu, Zeke Xie
While large language models (LLMs) have emerged as a significant advancement in artificial intelligence, the hardware and computational costs for training LLMs are also significant…