Compositional Performance Modeling of LLM Inference Clusters with Hierarchical KV Caching

Ruiyang Ma, Sheng Qi, DaiKang Kuang +2

Design-space exploration of LLM inference clusters requires predicting how hardware and serving policies interact with hierarchical key-value (KV) caching. Isolated operation profi…