3 papers
cs.LG2026
SpecMD: A Comprehensive Study On Speculative Expert Prefetching
Duc Hoang, Ajay Jaiswal, Mohammad Samragh +1
Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model's parameters is used during each inference. However, to translate this spar…
cs.LG2026
MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
Ajay Jaiswal, Lauren Hannah, Han-Byul Kim +4
Understanding how transformer components operate in LLMs is important, as it is at the core of recent technological advances in artificial intelligence. In this work, we revisit th…
cs.LG2025
Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
Ajay Jaiswal, Jianyu Wang, Yixiao Li +6
Sparsely activated Mixture-of-Experts (SMoE) has shown promise in scaling up the learning capacity of neural networks. However, vanilla SMoEs have issues such as expert redundancy…