Showing cs.CLShow all
2 papers · 1 filter
cs.CL2026
Emergent retokenization symmetry in large language models: phenomenology and applications
Kanishk Jain, Matthew Day, Tankut Can
Tokenization introduces representational redundancy: under a fixed token vocabulary, every byte string admits many valid token encodings, or segmentations, that decode to the same…
cs.CL2026
Semantic Chunking and the Entropy of Natural Language
Weishun Zhong, Doron Sivan, Tankut Can +2
The entropy rate of printed English is famously estimated to be about one bit per character, a benchmark that modern large language models (LLMs) have only recently approached. Thi…