Draft-based Approximate Inference for LLMs

Kevin Galim, Ethan Ewer, Wonjun Kang +3

Optimizing inference for long-context large language models (LLMs) is increasingly important due to the quadratic compute and linear memory cost of Transformers. Existing approxima…