Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
FG-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
Pingwei Sun, Yuxuan Hu, Jianchao Tan +6
Linear attention mechanisms have emerged as promising alternatives to softmax attention, offering linear-time complexity during inference. Recent advances such as Gated DeltaNet (G…
cs.LG2026
SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
Hongtao Xu, Jianchao Tan, Yuxuan Hu +8
While sparse attention mitigates the computational bottleneck of long-context LLM training, its distributed training process exhibits extreme heterogeneity in both \textit{1)} sequ…
cs.LG2026
WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
Jiacheng Li, Jianchao Tan, Zhidong Yang +11
Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architect…