The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving

Pai Zeng, Zhenyu Ning, Jieru Zhao +5

We survey the large language model (LLM) serving area to understand the intricate dynamics between cost-efficiency and accuracy, which is magnified by the growing need for longer c…