CUTE: Measuring LLMs' Understanding of Their Tokens

Lukas Edman, Helmut Schmid, Alexander Fraser

Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direc…