Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Higher-Dimensional Rotary Position Embedding
Yixing Li, Ruobing Xie, Yudong Zhang +3
Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotati…
cs.LG2025
TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
Yixing Li, Ruobing Xie, Zhen Yang +8
Transformers are the cornerstone of modern large language models, but their quadratic computational complexity limits efficiency in long-sequence processing. Recent advancements in…
cs.LG2025
Scaling Laws for Floating Point Quantization Training
Xingwu Sun, Shuaipeng Li, Ruobing Xie +13
Low-precision training is considered an effective strategy for reducing both training and downstream inference costs. Previous scaling laws for precision mainly focus on integer qu…