Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks

Vani Kanjirangat, Tanja Samardžić, Ljiljana Dolamic +1

Dialectal data are characterized by linguistic variation that appears small to humans but has a significant impact on the performance of models. This dialect gap has been related t…