ART: Attention Run-time Termination for Efficient Large Language Model Decoding

Chen Qiu, Guozhong Li, Cristian McGee +2

Long-context decoding in Large Language Models (LLMs) is constrained by the cost of accessing and processing the Key-Value (KV) cache. Despite evidence that attention outputs depen…