Comparative analysis of subword tokenization approaches for Indian languages

Sudhansu Bala Das, Samujjal Choudhury, Tapas Kumar Mishra +1

Tokenization is the act of breaking down text into smaller parts, or tokens, that are easier for machines to process. This is a key phase in machine translation (MT) models. Subwor…