PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications

Kuntai Du, Bowen Wang, Chen Zhang +9

Besides typical generative applications, like ChatGPT, GitHub Copilot, and Cursor, we observe an emerging trend that LLMs are increasingly used in traditional discriminative tasks,…