5 papers
vAttention: Verified Sparse Attention
Aditya Desai, Kumar Krishna Agrawal, Shuo Yang +5
State-of-the-art sparse attention methods for reducing decoding latency fall into two main categories: approximate top- (and its extension, top-) and recently introduced samp…
vCache: Verified Semantic Prompt Caching
Luis Gaspar Schroeder, Aditya Desai, Alejandro Cuadron +7
Semantic caches return cached responses for semantically similar prompts to reduce LLM inference latency and cost. They embed cached prompts and store them alongside their response…
Alto: Orchestrating Distributed Compound AI Systems with Nested Ancestry
Deepti Raghavan, Keshav Santhanam, Muhammad Shahir Rahman +7
Compound AI applications chain together subcomponents such as generative language models, document retrievers, and embedding models. Applying traditional systems optimizations such…
Optimizing LLM Queries in Relational Data Analytics Workloads
Shu Liu, Asim Biswal, Amog Kamsetty +8
Batch data analytics is a growing application for Large Language Models (LLMs). LLMs enable users to perform a wide range of natural language tasks, such as classification, entity…
The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks
Alejandro Cuadron, Dacheng Li, Wenjie Ma +13
Large Reasoning Models (LRMs) represent a breakthrough in AI problem-solving capabilities, but their effectiveness in interactive environments can be limited. This paper introduces…