TurboAttention: Efficient Attention Approximation For High Throughputs LLMs

Hao Kang, Srikant Bharadwaj, James Hensman +3

Large language model (LLM) inference demands significant amount of computation and memory, especially in the key attention mechanism. While techniques, such as quantization and acc…