4 papers
Understanding KV Residency in Production-Scale Agentic LLM Workloads
Mayank Arya, Agrim Bari, Anjaly Parayil +4
Agentic large language model (LLM) applications turn one user request into a sequence of model and tool calls over a growing context. Retaining this context's key-value (KV) state…
Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
Shashwat Jaiswal, Shrikara Arun, Anjaly Parayil +8
Low-Rank Adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of large language models (LLMs), enabling rapid adaptation to diverse domains. In prod…
Ensuring Fair LLM Serving Amid Diverse Applications
Redwan Ibne Seraj Khan, Kunal Jain, Haiying Shen +12
In a multi-tenant large language model (LLM) serving platform hosting diverse applications, some users may submit an excessive number of requests, causing the service to become una…
Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers
Rya Sanovar, Srikant Bharadwaj, Renee St. Amant +2
Transformer-based models have emerged as one of the most widely used architectures for natural language processing, natural language generation, and image generation. The size of t…