27 citations · 113 across the 24 of their papers we have counts for
24 papers
LLM-42: Enabling Determinism in LLM Inference with Verified Speculation
Raja Gond, Aditya K Kamath, Ramachandran Ramjee +1
In LLM inference, the same prompt may yield different outputs across different runs. At the system level, this non-determinism arises from floating-point non-associativity combined…
Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
Anish Biswas, Kanishk Goel, Srivarshinee S +5
Agentic applications are LLMs that iteratively invoke external tools to accomplish complex tasks. Such tool-based agents are rapidly becoming the dominant paradigm for deploying la…
Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1
Attention is the dominant source of latency during long-context LLM inference, an increasingly popular workload with reasoning models and RAG. We propose Kascade, a training-free s…
On Evaluating Performance of LLM Inference Serving Systems
Amey Agrawal, Nitin Kedia, Anmol Agarwal +5
The rapid evolution of Large Language Model (LLM) inference systems has yielded significant efficiency improvements. However, our systematic analysis reveals that current evaluatio…
TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference
Raja Gond, Nipun Kwatra, Ramachandran Ramjee
Distributed inference of large language models (LLMs) using tensor parallelism can introduce communication overheads of % even over GPUs connected via NVLink, a high-speed GPU…
Niyama : Breaking the Silos of LLM Inference Serving
Kanishk Goel, Jayashree Mohan, Nipun Kwatra +2
The widespread adoption of Large Language Models (LLMs) has enabled diverse applications with very different latency requirements. Existing LLM serving frameworks rely on siloed in…