Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention

Bin Gao, Zhuomin He, Puru Sharma +6

Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines executing multi-turn conver…