What do tokens know about their characters and how do they know it?

Ayush Kaushal, Kyle Mahowald

Pre-trained language models (PLMs) that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information, despite lacking explic…