Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
Sticky Routing: Training MoE Models for Memory-Efficient Inference
Ali Kayyam
Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping…
cs.LG2026
Do Transformers Need Three Projections? Systematic Study of QKV Variants
Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis
Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role. However, the individual cont…