6 papers
HiSpec: Hierarchical Speculative Decoding for LLMs
Avinash Kumar, Sujay Sanghavi, Poulami Das
Speculative decoding accelerates LLM inference by using a smaller draft model to speculate tokens that a larger target model verifies. Verification is often the bottleneck (e.g. ve…
GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
Sourish Wawdhane, Avinash Kumar, Poulami Das
Mixture-of-Expert (MoE) models enable efficient inference by employing smaller experts and activating only a subset of them per token. MoE serving engines distribute experts across…
Test-Time Speculation
Avinash Kumar, Sujay Sanghavi, Poulami Das
Speculative decoding accelerates LLM inference by using a fast draft model to generate tokens and a more accurate target model to verify them. Its performance depends on the $\text…
HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving
Avinash Kumar, Shashank Nag, Jason Clemons +2
Early-Exit Large Language Models (EE-LLMs) enable high throughput inference by allowing tokens to exit early at intermediate layers. However, their throughput is limited by the com…
Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs
Ravi Ghadia, Avinash Kumar, Gaurav Jain +2
Autoregressive Transformers rely on Key-Value (KV) caching to accelerate inference. However, the linear growth of the KV cache with context length leads to excessive memory consump…
Context Switching for Secure Multi-programming of Near-Term Quantum Computers
Avinash Kumar, Meng Wang, Chenxu Liu +3
Multi-programming quantum computers improve device utilization and throughput. However, crosstalk from concurrent two-qubit CNOT gates poses security risks, compromising the fideli…