Short-Context Dominance: How Much Local Context Natural Language Actually Needs?

Vala Vakilian, Zimeng Wang, Ankit Singh Rawat +1

We investigate the short-context dominance hypothesis: that for most sequences, a small local prefix suffices to predict their next tokens. Using large language models as statistic…