Learn Your Tokens: Word-Pooled Tokenization for Language Modeling

Avijit Thawani, Saurabh Ghanekar, Xiaoyuan Zhu +1

Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as 'ing' or w…