5 papers
How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee +9
Modern large language model (LLM) inference has progressively disaggregated to keep pace with growing model sizes and tight TTFT and TPOT service-level objectives: from chunked-pre…
Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6
Most recent state-of-the-art (SOTA) large language models (LLMs) use Mixture-of-Experts (MoE) architectures to scale model capacity without proportional per-token compute, enabling…
MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference
Abhimanyu Rajeshkumar Bambhaniya, Hanjiang Wu, Suvinay Subramanian +8
Modern LLM serving now spans multi-stage pipelines including RAG retrieval and KV cache reuse, each with distinct compute, memory, and latency demands. Inference engines expose a l…
Enabling Unstructured Sparse Acceleration on Structured Sparse Accelerators
Geonhwa Jeong, Po-An Tsai, Abhimanyu R. Bambhaniya +2
Exploiting sparsity in deep neural networks (DNNs) has been a promising area for meeting the growing computation requirements. To minimize the overhead of sparse acceleration, hard…
Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
Abhimanyu Bambhaniya, Ritik Raj, Geonhwa Jeong +6
Large language models (LLMs) have shown remarkable performance across a wide range of applications, often outperforming human experts. However, deploying these gigantic models effi…