How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers

Jonathan Roberts, Kai Han, Samuel Albanie

Frontier LLMs are increasingly utilised across academia, society and industry. A commonly used unit for comparing models, their inputs and outputs, and estimating inference pricing…