5 papers
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li +1
As real-world applications increasingly require processing inputs of 100k+ tokens, the gap between context length and inference efficiency has become a critical bottleneck. Context…
Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models
Shubhangi Upasani, Ravi Shanker Raju, Bo Li +5
Prompt length is a major bottleneck in agentic large language model (LLM) workloads, where repeated inference steps and multi-call loops incur substantial prefill cost. Recent work…
The Limits of Long-Context Reasoning in Automated Bug Fixing
Ravi Raju, Mengmeng Ji, Shubhangi Upasani +2
Rapidly increasing context lengths have led to the assumption that large language models (LLMs) can directly reason over entire codebases. Concurrently, recent advances in LLMs hav…
Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
Fenglu Hong, Ravi Raju, Jonathan Lingjie Li +5
Speculative decoding is an effective method for accelerating inference of large language models (LLMs) by employing a small draft model to predict the output of a target model. How…
Composition of Experts: A Modular Compound AI System Leveraging Large Language Models
Swayambhoo Jain, Ravi Raju, Bo Li +8
Large Language Models (LLMs) have achieved remarkable advancements, but their monolithic nature presents challenges in terms of scalability, cost, and customization. This paper int…