1 paper · 1 filter
Vaibhav Rathore, Siddhant Gole, Dadhichi Telwadkar +5
Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes. This is harmful for morpho…