Emergent retokenization symmetry in large language models: phenomenology and applications

Kanishk Jain, Matthew Day, Tankut Can

Tokenization introduces representational redundancy: under a fixed token vocabulary, every byte string admits many valid token encodings, or segmentations, that decode to the same…