Tokenisation via Convex Relaxations

Jan Tempus, Philip Whittington, Craig W. Schmidt +2

Tokenisation is an integral part of the current NLP pipeline. Current tokenisation algorithms such as BPE and Unigram are greedy algorithms -- they make locally optimal decisions w…