8 papers
LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs
Benno Krojer, Shravan Nayak, Oscar Mañas +4
Transforming a large language model (LLM) into a vision-language model (VLM) can be achieved by mapping the visual tokens from a vision encoder into the embedding space of an LLM.…
Back to Bytes: Revisiting Tokenization Through UTF-8
Amit Moryossef, Clara Meister, Pavel Stepachev +1
We present UTF8Tokenizer, a minimalist byte-level tokenizer that maps text exactly to IDs corresponding to the bytes underlying the text's UTF-8 encoding (e.g., byte x09 is token I…
Efficient Test-Time Scaling for Small Vision-Language Models
Mehmet Onurcan Kaya, Desmond Elliott, Dim P. Papadopoulos
Small Vision-Language Models (VLMs) provide a computationally efficient alternative to larger models, at the cost of weaker generalization abilities and downstream task performance…
Dynaword: From One-shot to Continuously Developed Datasets
Kenneth Enevoldsen, Kristian Nørgaard Jensen, Jan Kostkan +14
Large-scale datasets are foundational for research and development in natural language processing. However, current approaches face three key challenges: (1) reliance on ambiguousl…
Multilingual Pretraining for Pixel Language Models
Ilker Kesen, Jonas F. Lotz, Ingo Ziegler +2
Pixel language models operate directly on images of rendered text, eliminating the need for a fixed vocabulary. While these models have demonstrated strong capabilities for downstr…
PHD: Pixel-Based Language Modeling of Historical Documents
Nadav Borenstein, Phillip Rust, Desmond Elliott +1
The digitisation of historical documents has provided historians with unprecedented research opportunities. Yet, the conventional approach to analysing historical documents involve…
Text Rendering Strategies for Pixel Language Models
Jonas F. Lotz, Elizabeth Salesky, Phillip Rust +1
Pixel-based language models process text rendered as images, which allows them to handle any script, making them a promising approach to open vocabulary language modelling. However…
Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models
Laura Cabello, Emanuele Bugliarello, Stephanie Brandl +1
Pretrained machine learning models are known to perpetuate and even amplify existing biases in data, which can result in unfair outcomes that ultimately impact user experience. The…