6 papers
SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators
Jonathan Li, Nasim Farahini, Evgenii Iuliugin +19
The proliferation of 100B+ parameter Large Language Models (LLMs) with 100k+ context length support have resulted in increasing demands for on-chip memory to support large KV cache…
Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models
Qizheng Zhang, Changran Hu, Shubhangi Upasani +10
Large language model (LLM) applications such as agents and domain-specific reasoning increasingly rely on context adaptation: modifying inputs with instructions, strategies, or evi…
Test-Time Adaptation via Many-Shot Prompting: Benefits, Limits, and Pitfalls
Shubhangi Upasani, Chen Wu, Jay Rainton +4
Test-time adaptation enables large language models (LLMs) to modify their behavior at inference without updating model parameters. A common approach is many-shot prompting, where l…
Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models
Shubhangi Upasani, Ravi Shanker Raju, Bo Li +5
Prompt length is a major bottleneck in agentic large language model (LLM) workloads, where repeated inference steps and multi-call loops incur substantial prefill cost. Recent work…
The Limits of Long-Context Reasoning in Automated Bug Fixing
Ravi Raju, Mengmeng Ji, Shubhangi Upasani +2
Rapidly increasing context lengths have led to the assumption that large language models (LLMs) can directly reason over entire codebases. Concurrently, recent advances in LLMs hav…
LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
Guangtao Wang, Shubhangi Upasani, Chen Wu +5
Efficient long-context inference is critical as large language models (LLMs) adopt context windows of ranging from 128K to 1M tokens. However, the growing key-value (KV) cache and…