Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?

Ahmed Alajrami, Katerina Margatina, Nikolaos Aletras

Understanding how and what pre-trained language models (PLMs) learn about language is an open challenge in natural language processing. Previous work has focused on identifying whe…