Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
Junlin Chen, Daize Dong, Huanwei Di +9
BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are widely trusted. When we used FlashAttention-3 to…
cs.LG2025
Towards Analyzing and Understanding the Limitations of VAPO: A Theoretical Perspective
Jintian Shao, Yiming Cheng, Hongyi Huang +4
The VAPO framework has demonstrated significant empirical success in enhancing the efficiency and reliability of reinforcement learning for long chain-of-thought (CoT) reasoning ta…
cs.LG2025
ComplexFormer: Disruptively Advancing Transformer Inference Ability via Head-Specific Complex Vector Attention
Jintian Shao, Hongyi Huang, Jiayi Wu +4
Transformer models rely on self-attention to capture token dependencies but face challenges in effectively integrating positional information while allowing multi-head attention (M…