2 papers
cs.CL2026
TokenPilot: Cache-Efficient Context Management for LLM Agents
Buqiang Xu, Zirui Xue, Dianmou Chen +12
As LLM agents are deployed in long-horizon sessions, context accumulation drives up inference costs. Existing approaches utilize text pruning or dynamic memory eviction to minimize…
cs.DC2024
KVDirect: Distributed Disaggregated LLM Inference
Shiyang Chen, Rain Jiang, Dezhi Yu +6
Large Language Models (LLMs) have become the new foundation for many applications, reshaping human society like a storm. Disaggregated inference, which separates prefill and decode…