3 papers
cs.LG2026
Sticky Routing: Training MoE Models for Memory-Efficient Inference
Ali Kayyam
Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping…
cs.LG2026
Do Transformers Need Three Projections? Systematic Study of QKV Variants
Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis
Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role. However, the individual cont…
cs.CV2025
Provenance Networks: End-to-End Exemplar-Based Explainability
Ali Kayyam, Anusha Madan Gopal, M. Anthony Lewis
We introduce provenance networks, a novel class of neural models designed to provide end-to-end, training-data-driven explainability. Unlike conventional post-hoc methods, provenan…