XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference

João Monteiro, Étienne Marcotte, Pierre-André Noël +5

In-context learning (ICL) approaches typically leverage prompting to condition decoder-only language model generation on reference information. Just-in-time processing of a context…