5 papers
Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
Tiancheng Hu, Benjamin Minixhofer, Nigel Collier
The "alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliab…
Bolmo: Byteifying the Next Generation of Language Models
Benjamin Minixhofer, Tyler Murray, Tomasz Limisiewicz +6
Recent advances in generative AI have been largely driven by large language models (LLMs), deep neural networks that operate over discrete units called tokens. To represent text, t…
Zero-Shot Tokenizer Transfer
Benjamin Minixhofer, Edoardo Maria Ponti, Ivan VuliÄ
Language models (LMs) are bound to their tokenizer, which maps raw text to a sequence of vocabulary items (tokens). This restricts their flexibility: for example, LMs trained prima…
Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching
Benjamin Minixhofer, Ivan VuliÄ, Edoardo Maria Ponti
Distillation has shown remarkable success in transferring knowledge from a Large Language Model (LLM) teacher to a student LLM. However, current distillation methods require simila…
Retrofitting Large Language Models with Dynamic Tokenization
Darius Feher, Ivan VuliÄ, Benjamin Minixhofer
Current language models (LMs) use a fixed, static subword tokenizer. This default choice typically results in degraded efficiency and language capabilities, especially in languages…