Self-Selected Attention Span for Accelerating Large Language Model Inference

Tian Jin, Wanzin Yazar, Zifei Xu +2

Large language models (LLMs) can solve challenging tasks. However, their inference computation on modern GPUs is highly inefficient due to the increasing number of tokens they must…