No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance

Antonio Lopardo, Luis Frentzen Salim, Shikhar Srivastava +3

Tokenization is a critical component of the language modeling pipeline, but it is poorly understood. Tokenizers can impact downstream performance, but there is currently mixed evid…