Back to Bytes: Revisiting Tokenization Through UTF-8

Amit Moryossef, Clara Meister, Pavel Stepachev +1

We present UTF8Tokenizer, a minimalist byte-level tokenizer that maps text exactly to IDs corresponding to the bytes underlying the text's UTF-8 encoding (e.g., byte x09 is token I…