From Language Models over Tokens to Language Models over Characters

Tim Vieira, Ben LeBrun, Mario Giulianelli +5

Modern language models are internally -- and mathematically -- distributions over $token$ strings rather than $character$ strings, posing numerous challenges for programm…