3 papers
cs.LG2026
Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path
Victor Norgren
A stateless inference server (vLLM, SGLang, TensorRT-LLM) idles between requests while the accelerator waits; a stateful session reclaims that idle time. Speculative pre-positionin…
cs.LG2026
Stateful Inference for Low-Latency Multi-Agent Tool Calling
Victor Norgren
Multi-agent tool calling is becoming the dominant interaction pattern for LLM-based systems, yet existing inference frameworks treat each tool call as an independent request, re-pr…
cs.LG2026
Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers
Victor Norgren
Conventional transformer inference engines are request-driven, paying an O(n) prefill cost on every query. In streaming workloads, where data arrives continuously and queries probe…