Where is the signal in tokenization space?

Renato Lui Geh, Honghua Zhang, Kareem Ahmed +2

Large Language Models (LLMs) are typically shipped with tokenizers that deterministically encode text into so-called canonical token sequences, to which the LLMs assign probability…