Showing 2026 · cs.CLShow all
2 papers · 2 filters
cs.CL2026
LangMAP: A Language-Adaptive Approach to Tokenization
Clara Meister, Suchir Salhan, Andrzej Szablewski +3
Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. Using one with an existing pretrained model requires either t…
cs.CL2026
Tokenisation via Convex Relaxations
Jan Tempus, Philip Whittington, Craig W. Schmidt +2
Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms -- they make locally optimal decisions w…