When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

Haonan Wang, Qian Liu, Chao Du +4

Extending context window sizes allows large language models (LLMs) to process longer sequences and handle more complex tasks. Rotary Positional Embedding (RoPE) has become the de f…