FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines

Jiaao He, Jidong Zhai

Cost of serving large language models (LLM) is high, but the expensive and scarce GPUs are poorly efficient when generating tokens sequentially, unless the batch of sequences is en…