Learning When Not to Listen: Selective Anti-Interference Pretraining for Language Models

Jinchang Zhu, Haowei He, Yi Ding +5

Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated prefix tokens are perturbed…