Showing 2026Show all
3 papers · 1 filter
cs.CL2026
MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers
Linrui Ma, Chun Hei Lo, Xinyu Wang +12
The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particula…
cs.LG2026
LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
Wenkai Chen, Tianshu Li, Wenyong Huang +3
Mixture-of-Experts (MoE) and looped architectures scale models along two orthogonal axes, namely parameter capacity and effective depth. However, mainstream looped architectures re…
cs.LG2026
UniPool: Learning Expert-to-Layer Ownership from Brief Global Access
Minbin Huang, Han Shi, Chuanyang Zheng +5
Most Mixture-of-Experts (MoE) transformers preassign each expert to one layer for the whole of training. We study whether a brief global-access phase can learn a compatible expert-…