4 papers
TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference
Raja Gond, Nipun Kwatra, Ramachandran Ramjee
Distributed inference of large language models (LLMs) using tensor parallelism can introduce communication overheads of % even over GPUs connected via NVLink, a high-speed GPU…
Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1
Attention is the dominant source of latency during long-context LLM inference, an increasingly popular workload with reasoning models and RAG. We propose Kascade, a training-free s…
On Evaluating Performance of LLM Inference Serving Systems
Amey Agrawal, Nitin Kedia, Anmol Agarwal +5
The rapid evolution of Large Language Model (LLM) inference systems has yielded significant efficiency improvements. However, our systematic analysis reveals that current evaluatio…
Niyama : Breaking the Silos of LLM Inference Serving
Kanishk Goel, Jayashree Mohan, Nipun Kwatra +2
The widespread adoption of Large Language Models (LLMs) has enabled diverse applications with very different latency requirements. Existing LLM serving frameworks rely on siloed in…