On the Effect of (Near) Duplicate Subwords in Language Modelling

Anton Schäfer, Thomas Hofmann, Imanol Schlag +1

Tokenisation is a core part of language models (LMs). It involves splitting a character sequence into subwords which are assigned arbitrary indices before being served to the LM. W…