5 citations · 9 across the 11 of their papers we have counts for
11 papers
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Sangwhan Moon, Daisuke Oba, Youmi Ma +2
Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investi…
Bit-level BPE: Below the byte boundary
Sangwhan Moon, Tatsuya Hiraoka, Naoaki Okazaki
Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragma…
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs
LLM-jp, :, Akiko Aizawa +80
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and stron…
An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
Marco Cognetta, Tatsuya Hiraoka, Naoaki Okazaki +2
We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a postprocessing step that replaces rare subwords with their component subwords. The technique…
Knowledge of Pretrained Language Models on Surface Information of Tokens
Tatsuya Hiraoka, Naoaki Okazaki
Do pretrained language models have knowledge regarding the surface information of tokens? We examined the surface information stored in word or subword embeddings acquired by pretr…
Downstream Task-Oriented Neural Tokenizer Optimization with Vocabulary Restriction as Post Processing
Tatsuya Hiraoka, Tomoya Iwakura
This paper proposes a method to optimize tokenization for the performance improvement of already trained downstream models. Our method generates tokenization results attaining lowe…
Tokenization Preference for Human and Machine Learning Model: An Annotation Study
Tatsuya Hiraoka, Tomoya Iwakura
Is preferred tokenization for humans also preferred for machine-learning (ML) models? This study examines the relations between preferred tokenization for humans (appropriateness a…
MaxMatch-Dropout: Subword Regularization for WordPiece
Tatsuya Hiraoka
We present a subword regularization method for WordPiece, which uses a maximum matching algorithm for tokenization. The proposed method, MaxMatch-Dropout, randomly drops words in a…
Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation
Sho Takase, Tatsuya Hiraoka, Naoaki Okazaki
Subword regularizations use multiple subword segmentations during training to improve the robustness of neural machine translation models. In previous subword regularizations, we u…
Joint Optimization of Tokenization and Downstream Model
Tatsuya Hiraoka, Sho Takase, Kei Uchiumi +2
Since traditional tokenizers are isolated from a downstream task and model, they cannot output an appropriate tokenization depending on the task and model, although recent studies…
Named Entity Recognition and Relation Extraction using Enhanced Table Filling by Contextualized Representations
Youmi Ma, Tatsuya Hiraoka, Naoaki Okazaki
In this study, a novel method for extracting named entities and relations from unstructured text based on the table representation is presented. By using contextualized word embedd…