Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inference

Yaohua Tang, Zhicheng Hu, Kun Cheng +4

The increasing context window size in large language models (LLMs) has improved their ability to handle complex, long-text tasks. However, as the conversation rounds continue, it i…