Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
Kumari Nishu, Han-Byul Kim, Santosh Chilkunda +13
Mixture-of-Experts (MoE) models are increasingly deployed alongside Speculative Decoding (SD) to accelerate inference, but combining the two is challenging. SD improves the inferen…
cs.LG2026
MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
Ajay Jaiswal, Lauren Hannah, Han-Byul Kim +4
Understanding how transformer components operate in LLMs is important, as it is at the core of recent technological advances in artificial intelligence. In this work, we revisit th…
cs.LG2025
MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
Lauren. A Hannah, Soheil Zibakhsh, Kumari Nishu +4
Sparse Mixtures of Experts (MoEs) are typically trained to operate at a fixed sparsity level, e.g. in a top- gating function. This global sparsity level determines an operat…