KV Prediction for Improved Time to First Token

Maxwell Horton, Qingqing Cao, Chenfan Sun +4

Inference with transformer-based language models begins with a prompt processing step. In this step, the model generates the first output token and stores the KV cache needed for f…