When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models

Yefan Tao, Gerald Friedland, Luyang Kong

The performance of textual neural models often degrades when their inputs are corrupted by noise such as typos, OCR errors, or dropped words. We study the degradation rate across n…