Parallel Context-of-Experts Decoding for Retrieval Augmented Generation

Giulio Corallo, Paolo Papotti

Retrieval Augmented Generation faces a trade-off: concatenating documents in a long prompt enables multi-document reasoning but creates prefill bottlenecks, while encoding document…