4 papers · 1 filter
Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning
Liangyao Jin, Zhixin Wang, Liming Liu +8
Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous training. The resulting mismatch is especially harmful in…
Topology-Aware Revival for Efficient Sparse Training
Meiling Jin, Fei Wang, Xiaoyun Yuan +2
Static sparse training is a promising route to efficient learning by committing to a fixed mask pattern, yet the constrained structure reduces robustness. Early pruning decisions c…
Demystifying the Slash Pattern in Attention: The Role of RoPE
Yuan Cheng, Fengzhuo Zhang, Yunlong Hou +5
Large Language Models (LLMs) often exhibit slash attention patterns, where attention scores concentrate along the -th sub-diagonal for some offset . These patterns play a key…
Metis: Training LLMs with FP4 Quantization
Hengjie Cao, Mengyi Chen, Yifeng Yang +13
This work identifies anisotropy in the singular value spectra of parameters, activations, and gradients as the fundamental barrier to low-bit training of large language models (LLM…