Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE

Firoj Ahmmed Patwary, Abdullah Al Noman

Tokenization is an important first step in Natural Language Processing (NLP) pipelines because it decides how models learn and represent linguistic information. However, current su…