5 papers · 1 filter
Back to Bytes: Revisiting Tokenization Through UTF-8
Amit Moryossef, Clara Meister, Pavel Stepachev +1
We present UTF8Tokenizer, a minimalist byte-level tokenizer that maps text exactly to IDs corresponding to the bytes underlying the text's UTF-8 encoding (e.g., byte x09 is token I…
Dynaword: From One-shot to Continuously Developed Datasets
Kenneth Enevoldsen, Kristian Nørgaard Jensen, Jan Kostkan +14
Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguousl…
Multilingual Pretraining for Pixel Language Models
Ilker Kesen, Jonas F. Lotz, Ingo Ziegler +2
Pixel language models operate directly on images of rendered text, eliminating the need for a fixed vocabulary. While these models have demonstrated strong capabilities for downstr…
PHD: Pixel-Based Language Modeling of Historical Documents
Nadav Borenstein, Phillip Rust, Desmond Elliott +1
The digitisation of historical documents has provided historians with unprecedented research opportunities. Yet, the conventional approach to analysing historical documents involve…
Text Rendering Strategies for Pixel Language Models
Jonas F. Lotz, Elizabeth Salesky, Phillip Rust +1
Pixel-based language models process text rendered as images, which allows them to handle any script, making them a promising approach to open vocabulary language modelling. However…