1 citations · 1 across the 8 of their papers we have counts for
4 papers · 1 filter
Explicit Boundary Markers for Subword Vocabularies
Sander Land, Clara Meister
Subword tokenizers represent many common words twice in space-using writing systems, once with a leading space and once without. The two entries have separate embeddings in models,…
MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
Sander Land
The Unigram tokenizer uses an elegant representation which makes it straightforward to edit vocabularies, but its training is comparatively heavy and complex. We introduce MinGram…
Auditing LLM Benchmarks with Item Response Theory
Sander Land, Daniel M. Bikel
LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces li…
Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens
Zhenyu Zhao, Sander Land, Daniel M. Bikel +1
Reasoning in Large Language Models incurs significant inference-time compute, yet the token-level information structure of reasoning traces remains underexplored. We observe that r…