BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran +1

Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentatio…