Showing cs.CLShow all
2 papers · 1 filter
cs.CL2026
Tokenisation via Convex Relaxations
Jan Tempus, Philip Whittington, Craig W. Schmidt +2
Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms -- they make locally optimal decisions w…
cs.CL2025
Tokenisation over Bounded Alphabets is Hard
Violeta Kastreva, Philip Whittington, Dennis Komm +1
Recent works have shown that tokenisation is NP-complete. However, these works assume tokenisation is applied to inputs with unboundedly large alphabets -- an unrealistic assumptio…