S${}^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput

Yunho Jin, Chun-Feng Wu, David Brooks +1

Generating texts with a large language model (LLM) consumes massive amounts of memory. Apart from the already-large model parameters, the key/value (KV) cache that holds informatio…