Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens

Itay Itzhak, Omer Levy

Standard pretrained language models operate on sequences of subword tokens without direct access to the characters that compose each token's string representation. We probe the emb…