Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

Arda Uzunoglu, Benjamin Van Durme, Benjamin van Durme +1

Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories. This scaling reflects the implicit…