5 papers · 1 filter
Tokenizing Crosslingual Homographs
Rotem Brillant, Yuval Pinter
Multilingual language models rely on shared subword vocabularies to represent multiple languages within a limited number of token units. While such sharing is often useful, it can…
Inside the LLM Word Factory
Benzi Busigin, Yuval Pinter
Transformer language models process input provided as subword fragments, but natural language semantics usually rely on word-level concepts. Detokenization is the process where mod…
Which Pieces Does Unigram Tokenization Really Need?
Sander Land, Yuval Pinter
The Unigram tokenization algorithm offers a probabilistic alternative to the greedy heuristics of Byte-Pair Encoding. Despite its theoretical elegance, its implementation in practi…
CharBench: Evaluating the Role of Tokenization in Character-Level Tasks
Omri Uzan, Yuval Pinter
Tasks that require character-level reasoning, such as counting or locating characters within words, remain challenging for contemporary language models. A common conjecture is that…
Hebrew Diacritics Restoration using Visual Representation
Yair Elboher, Yuval Pinter
Diacritics restoration in Hebrew is a fundamental task for ensuring accurate word pronunciation and disambiguating textual meaning. Despite the language's high degree of ambiguity…