most citedHow do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese

2 citations · 2 across the 1 of their papers we have counts for

collaborators

1 paper

cs.CL2023★ 2 cited

How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese

Takuro Fujii, Koki Shibata, Atsuki Yamaguchi +2

This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist be…