7 papers
SETA: Scaling Environments for Terminal Agents
Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru +19
Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interfaces (GUIs). Among these, the te…
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li +1
As real-world applications increasingly require processing inputs of 100k+ tokens, the gap between context length and inference efficiency has become a critical bottleneck. Context…
SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators
Jonathan Li, Nasim Farahini, Evgenii Iuliugin +19
The proliferation of 100B+ parameter Large Language Models (LLMs) with 100k+ context length support have resulted in increasing demands for on-chip memory to support large KV cache…
Synthetic Document Question Answering in Hungarian
Jonathan Li, Zoltan Csaki, Nidhi Hiremath +4
Modern VLMs have achieved near-saturation accuracy in English document visual question-answering (VQA). However, this task remains challenging in lower resource languages due to a…
Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
Fenglu Hong, Ravi Raju, Jonathan Lingjie Li +5
Speculative decoding is an effective method for accelerating inference of large language models (LLMs) by employing a small draft model to predict the output of a target model. How…
LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
Guangtao Wang, Shubhangi Upasani, Chen Wu +5
Efficient long-context inference is critical as large language models (LLMs) adopt context windows of ranging from 128K to 1M tokens. However, the growing key-value (KV) cache and…