69 citations · 113 across the 12 of their papers we have counts for
12 papers
A Family of LLMs Liberated from Static Vocabularies
Aleph Alpha, :, Adnen Abdessaied +35
Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although lea…
Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs
Mehdi Ali, Michael Fromm, Klaudia Thellmann +38
We present two multilingual LLMs, Teuken 7B-base and Teuken 7B-instruct, designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European U…
u-P: The Unit-Scaled Maximal Update Parametrization
Charlie Blake, Constantin Eichenberg, Josef Dean +7
The Maximal Update Parametrization (P) aims to make the optimal hyperparameters (HPs) of a model independent of its size, allowing them to be swept using a cheap proxy model rat…
T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
Björn Deiseroth, Manuel Brack, Patrick Schramowski +2
Tokenizers are crucial for encoding information in Large Language Models, but their development has recently stagnated, and they contain inherent weaknesses. Major limitations incl…
Efficient Parallelization Layouts for Large-Scale Distributed Model Training
Johannes Hagemann, Samuel Weinbach, Konstantin Dobler +2
Efficiently training large language models requires parallelizing across hundreds of hardware accelerators and invoking various compute and memory optimizations. When combined, man…
Tokenizer Choice For LLM Training: Negligible or Crucial?
Mehdi Ali, Michael Fromm, Klaudia Thellmann +18
The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and…
MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
Marco Bellagente, Manuel Brack, Hannah Teufel +11
The recent popularity of text-to-image diffusion models (DM) can largely be attributed to the intuitive interface they provide to users. The intended generation can be expressed in…
AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation
Björn Deiseroth, Mayukh Deb, Samuel Weinbach +3
Generative transformer models have become increasingly complex, with large numbers of parameters and the ability to process multiple input modalities. Current methods for explainin…
M-VADER: A Model for Diffusion with Multimodal Context
Samuel Weinbach, Marco Bellagente, Constantin Eichenberg +7
We introduce M-VADER: a diffusion model (DM) for image generation where the output can be specified using arbitrary combinations of images and text. We show how M-VADER enables the…
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Sid Black, Stella Biderman, Eric Hallahan +14
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a…
MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning
Constantin Eichenberg, Sidney Black, Samuel Weinbach +2
Large-scale pretraining is fast becoming the norm in Vision-Language (VL) modeling. However, prevailing VL approaches are limited by the requirement for labeled data and the use of…
Domain-Level Explainability -- A Challenge for Creating Trust in Superhuman AI Strategies
Jonas Andrulis, Ole Meyer, Grégory Schott +2
For strategic problems, intelligent systems based on Deep Reinforcement Learning (DRL) have demonstrated an impressive ability to learn advanced solutions that can go far beyond hu…