4 papers
A Family of LLMs Liberated from Static Vocabularies
Aleph Alpha, :, Adnen Abdessaied +35
Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although lea…
SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models
Pit Neitemeier, Alessio Serra, Jiaze Li +3
Hierarchical sequence models replace fixed tokenization with learned segmentations that compress long byte sequences for efficient autoregressive modeling. While recent end-to-end…
Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn +2
Depth-recurrence facilitates latent reasoning by sharing parameters across depths. However, prior work lacks combined FLOP-, parameter-, and memory-matched baselines, underutilizes…
Single-Pass Object-Focused Data Selection
Niclas Popp, Dan Zhang, Jan Hendrik Metzen +2
While unlabeled image data is often plentiful, the costs of high-quality labels pose an important practical challenge: Which images should one select for labeling to use the annota…