1 citations · 2 across the 7 of their papers we have counts for
3 papers · 1 filter
Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights
Maharaj Brahma, N J Karthika, Rajat Verma +4
Tokenization plays a pivotal role in NLP and is fundamental to training language models. However, existing tokenizers are often skewed towards high-resource languages, limiting the…
MorphTok: Morphologically Grounded Tokenization for Indian Languages
Maharaj Brahma, N J Karthika, Atul Singh +5
Tokenization is a crucial step in NLP, especially with the rise of large language models (LLMs), impacting downstream performance, computational cost, and efficiency. Existing LLMs…
Semantically Cohesive Word Grouping in Indian Languages
N J Karthika, Adyasha Patra, Nagasai Saketh Naidu +3
Indian languages are inflectional and agglutinative and typically follow clause-free word order. The structure of sentences across most major Indian languages are similar when thei…