Do Long-Range Language Models Actually Use Long-Range Context?

Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke +1

Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve th…