Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models

Sangwhan Moon, Daisuke Oba, Youmi Ma +2

Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investi…