2 citations · 8 across the 23 of their papers we have counts for
8 papers · 1 filter
KV-Kaizen: Learning Context-Adaptive Cache Compression Choices
Joao Monteiro, Louis Béthune, Anastasiia Filippova +3
As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This impacts LLM throughput negativ…
On Trajectory-Aware Training for Masked Diffusion Language Models
Manuel Madeira, Amitis Shidani, Alice Bizeul +6
Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The model is trained on randomly mas…
Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
Paul Jeha, Anastasiia Sedova, Louis Béthune +4
For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization…
DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
Eleonora Gualdoni, Sonia Laguna, Louis Bethune +3
Multi-domain fine-tuning of large language models requires improving performance on target domains while preserving previous capabilities, such as general knowledge, instruction fo…
Scaling Categorical Flow Maps
Oscar Davis, Anastasiia Filippova, Pierre Ablin +4
Continuous diffusion and flow matching models could represent a powerful alternative to autoregressive approaches for language modelling (LM), as they unlock a host of advantages c…
Optimal Splitting of Language Models from Mixtures to Specialized Domains
Skyler Seto, Pierre Ablin, Anastasiia Filippova +4
Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard t…
The Design Space of Tri-Modal Masked Diffusion Models
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec +21
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal gen…
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss
Szilvia Ujváry, Louis Béthune, Pierre Ablin +3
Language models have consistently grown to compress more world knowledge into their parameters, but the knowledge that can be pretrained into them is upper-bounded by their paramet…