efficient training 1large language models 1residual stream expansion 1sparse computation 1transformer architectures 1
From the 1 of 2 linked papers with an AI index.
2 papers
cs.LG2026
xHC: Expanded Hyper-Connections
Xiangdong Zhang, Xiaohan Qin, Sunan Zou +10
The paper introduces xHC, a method that expands the residual stream of Transformers to many parallel streams using temporal feature augmentation and a sparse update scheme, enablin…
cs.LG2026
JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation
Yebin Yang, Huaijin Wu, Fu Guo +5
LLMs have traditionally scaled along dense dimensions, where performance is coupled with near-linear increases in computational cost. While MoE decouples capacity from compute, it…