Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving

Zhixin Wang, Zhengbo Wang, Fangcheng Fu +15

Heterogeneous prefill-decode (PD) inference is now in production: prefill on cost-efficient or supply-available accelerators, decode on bandwidth-strong ones, and KV state crossing…