12 citations · 16 across the 6 of their papers we have counts for
5 papers · 1 filter
Cacheback: Speculative Decoding With Nothing But Cache
Zhiyao Ma, In Gim, Lin Zhong
We present Cacheback Decoding, a training-free and model-agnostic speculative decoding method that exploits the locality in language to accelerate Large Language Model (LLM) infere…
Serve Programs, Not Prompts
In Gim, Lin Zhong
Current large language model (LLM) serving systems, primarily designed for text completion, are neither efficient nor adaptable for increasingly complex LLM applications due to the…
Pie: A Programmable Serving System for Emerging LLM Applications
In Gim, Zhiyao Ma, Seung-seob Lee +1
Emerging large language model (LLM) applications involve diverse reasoning strategies and agentic workflows, straining the capabilities of existing serving systems built on a monol…
Asynchronous LLM Function Calling
In Gim, Seung-seob Lee, Lin Zhong
Large language models (LLMs) use function calls to interface with external tools and data source. However, the current approach to LLM function calling is inherently synchronous, w…
Prompt Cache: Modular Attention Reuse for Low-Latency Inference
In Gim, Guojun Chen, Seung-seob Lee +3
We present Prompt Cache, an approach for accelerating inference for large language models (LLM) by reusing attention states across different LLM prompts. Many input prompts have ov…