KVDirect: Distributed Disaggregated LLM Inference

Shiyang Chen, Rain Jiang, Dezhi Yu +6

Large Language Models (LLMs) have become the new foundation for many applications, reshaping human society like a storm. Disaggregated inference, which separates prefill and decode…