SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance

Rya Sanovar, Srikant Bharadwaj, Hritvik Taneja +1

Retrieval-Augmented Generation (RAG) injects LLM queries with relevant documents to improve response quality. This injection increases prompt length and slows time to first token (…