most citedPerformance Evaluation of Tokenizers in Large Language Models for the Assamese Language

3 citations

researchers with a paper here

Counting papers with at most 25 authors, so collaboration author lists do not stand in for the people here.

collaborating institutions
Showing cs.CLShow all

1 paper · 1 filter

cs.CL2024★ 3 cited

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language

Sagar Tamang, Dibya Jyoti Bora

Training of a tokenizer plays an important role in the performance of deep learning models. This research aims to understand the performance of tokenizers in five state-of-the-art…