6 papers
Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
Junlin Chen, Daize Dong, Huanwei Di +9
BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are widely trusted. When we used FlashAttention-3 to…
IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models
Yiqi Wang, Zhifeng Rao, Jiaqi Zhang +7
Open robot policies increasingly follow two paradigms: vision-language-action models (VLAs) directly map observations and instructions to actions, whereas world-action models (WAMs…
Towards Analyzing and Understanding the Limitations of VAPO: A Theoretical Perspective
Jintian Shao, Yiming Cheng, Hongyi Huang +4
The VAPO framework has demonstrated significant empirical success in enhancing the efficiency and reliability of reinforcement learning for long chain-of-thought (CoT) reasoning ta…
ComplexFormer: Disruptively Advancing Transformer Inference Ability via Head-Specific Complex Vector Attention
Jintian Shao, Hongyi Huang, Jiayi Wu +4
Transformer models rely on self-attention to capture token dependencies but face challenges in effectively integrating positional information while allowing multi-head attention (M…
VQ-Logits: Compressing the Output Bottleneck of Large Language Models via Vector Quantized Logits
Jintian Shao, Hongyi Huang, Jiayi Wu +4
Large Language Models (LLMs) have achieved remarkable success but face significant computational and memory challenges, particularly due to their extensive output vocabularies. The…
SAU: A Dual-Branch Network to Enhance Long-Tailed Recognition via Generative Models
Guangxi Li, Yinsheng Song, Mingkai Zheng
Long-tailed distributions in image recognition pose a considerable challenge due to the severe imbalance between a few dominant classes with numerous examples and many minority cla…