AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

Xin Wang, Ziming Miao, Yi Zhu +4

Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents w…