7 citations · 25 across the 10 of their papers we have counts for
10 papers
A Family of LLMs Liberated from Static Vocabularies
Aleph Alpha, :, Adnen Abdessaied +35
Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although lea…
1-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization
Sohir Maskey, Constantin Eichenberg, Johannes Messner +1
Quantization-aware training (QAT) is an effective method to drastically reduce the memory footprint of LLMs while keeping performance degradation at an acceptable level. However, t…
Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models
Pit Neitemeier, Björn Deiseroth, Constantin Eichenberg +1
Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tokenizers have become the…
u-P: The Unit-Scaled Maximal Update Parametrization
Charlie Blake, Constantin Eichenberg, Josef Dean +7
The Maximal Update Parametrization (P) aims to make the optimal hyperparameters (HPs) of a model independent of its size, allowing them to be swept using a cheap proxy model rat…
Divergent Token Metrics: Measuring degradation to prune away LLM components -- and optimize quantization
Björn Deiseroth, Max Meuer, Nikolas Gritsch +4
Large Language Models (LLMs) have reshaped natural language processing with their impressive capabilities. However, their ever-increasing size has raised concerns about their effec…
MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
Marco Bellagente, Manuel Brack, Hannah Teufel +11
The recent popularity of text-to-image diffusion models (DM) can largely be attributed to the intuitive interface they provide to users. The intended generation can be expressed in…
M-VADER: A Model for Diffusion with Multimodal Context
Samuel Weinbach, Marco Bellagente, Constantin Eichenberg +7
We introduce M-VADER: a diffusion model (DM) for image generation where the output can be specified using arbitrary combinations of images and text. We show how M-VADER enables the…
MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning
Constantin Eichenberg, Sidney Black, Samuel Weinbach +2
Large-scale pretraining is fast becoming the norm in Vision-Language (VL) modeling. However, prevailing VL approaches are limited by the requirement for labeled data and the use of…
Self-similar behavior of the exchange-driven growth model with product kernel
Constantin Eichenberg, André Schlichting
We study the self-similar behavior of the exchange-driven growth model, which describes a process in which pairs of clusters, consisting of an integer number of monomers, interact…
Special solutions to a non-linear coarsening model with local interactions
Constantin Eichenberg
We consider a class of mass transfer models on a one-dimensional lattice with nearest-neighbour interactions. The evolution is given by the discrete backward fast diffusion equatio…