2 papers
cs.CL2026
Distilling Token-Trained Models into Byte-Level Models
Zishuo Bao, Jiaqi Leng, Junxiong Wang +2
Byte Language Models (BLMs) have emerged as a promising direction for scaling language models beyond tokenization. However, existing BLMs typically require training from scratch on…
cs.LG2024
DeMo: Decoupled Momentum Optimization
Bowen Peng, Lizhang Chen, Baiyu Su +3
Scaling neural network training increasingly depends on synchronous data-parallelism, yet full-precision gradient all-reduce imposes a severe communication bottleneck. We propose D…