4 papers
SpecMD: A Comprehensive Study On Speculative Expert Prefetching
Duc Hoang, Ajay Jaiswal, Mohammad Samragh +1
Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model's parameters is used during each inference. However, to translate this spar…
MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE
Soheil Zibakhsh, Mohammad Samragh, Kumari Nishu +3
The generation quality of large language models (LLMs) is often improved by utilizing inference-time sequence-level scaling methods (e.g., Chain-of-Thought). We introduce hyper-par…
Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
Mohammad Samragh, Arnav Kundu, David Harrison +4
Autoregressive language models are constrained by their inherently sequential nature, generating one token at a time. This paradigm limits inference speed and parallelism, especial…
SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models
Han-Byul Kim, Duc Hoang, Arnav Kundu +2
With the rapid expansion in the scale of large language models (LLMs), enabling efficient distributed inference across multiple computing units has become increasingly critical. Ho…