5 papers
Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
Kumari Nishu, Han-Byul Kim, Santosh Chilkunda +13
Mixture-of-Experts (MoE) models are increasingly deployed alongside Speculative Decoding (SD) to accelerate inference, but combining the two is challenging. SD improves the inferen…
TIDE: Every Layer Knows the Token Beneath the Context
Ajay Jaiswal, Lauren Hannah, Han-Byul Kim +3
We revisit a universally accepted but under-examined design choice in every modern LLM: a token index is looked up once at the input embedding layer and then permanently discarded.…
MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
Ajay Jaiswal, Lauren Hannah, Han-Byul Kim +4
Understanding how transformer components operate in LLMs is important, as it is at the core of recent technological advances in artificial intelligence. In this work, we revisit th…
MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE
Soheil Zibakhsh, Mohammad Samragh, Kumari Nishu +3
The generation quality of large language models (LLMs) is often improved by utilizing inference-time sequence-level scaling methods (e.g., Chain-of-Thought). We introduce hyper-par…
MoE-PHDS: One MoE checkpoint for flexible runtime sparsity
Lauren. A Hannah, Soheil Zibakhsh, Kumari Nishu +4
Sparse Mixtures of Experts (MoEs) are typically trained to operate at a fixed sparsity level, e.g. in a top- gating function. This global sparsity level determines an operat…