Showing 2023Show all
2 papers · 1 filter
cs.CL2023
Downstream Task-Oriented Neural Tokenizer Optimization with Vocabulary Restriction as Post Processing
Tatsuya Hiraoka, Tomoya Iwakura
This paper proposes a method to optimize tokenization for the performance improvement of already trained downstream models. Our method generates tokenization results attaining lowe…
cs.CL2023
Tokenization Preference for Human and Machine Learning Model: An Annotation Study
Tatsuya Hiraoka, Tomoya Iwakura
Is preferred tokenization for humans also preferred for machine-learning (ML) models? This study examines the relations between preferred tokenization for humans (appropriateness a…