Block-Attention for Efficient Prefilling

Dongyang Ma, Yan Wang, Lan Tian

We introduce Block-attention, an attention mechanism designed to address the increased inference latency and cost in Retrieval-Augmented Generation (RAG) scenarios. Traditional app…