Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness

Zhipeng Yang, Shu Yang, Lijie Hu +1

Large language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms unde…