Joint Optimization of Tokenization and Downstream Model

Tatsuya Hiraoka, Sho Takase, Kei Uchiumi +2

Since traditional tokenizers are isolated from a downstream task and model, they cannot output an appropriate tokenization depending on the task and model, although recent studies…