2 citations · 8 across the 23 of their papers we have counts for
30 papers
KV-Kaizen: Learning Context-Adaptive Cache Compression Choices
Joao Monteiro, Louis Béthune, Anastasiia Filippova +3
As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This impacts LLM throughput negativ…
On Trajectory-Aware Training for Masked Diffusion Language Models
Manuel Madeira, Amitis Shidani, Alice Bizeul +6
Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The model is trained on randomly mas…
Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
Paul Jeha, Anastasiia Sedova, Louis Béthune +4
For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization…
DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
Eleonora Gualdoni, Sonia Laguna, Louis Bethune +3
Multi-domain fine-tuning of large language models requires improving performance on target domains while preserving previous capabilities, such as general knowledge, instruction fo…
Scaling Categorical Flow Maps
Oscar Davis, Anastasiia Filippova, Pierre Ablin +4
Continuous diffusion and flow matching models could represent a powerful alternative to autoregressive approaches for language modelling (LM), as they unlock a host of advantages c…
Optimal Splitting of Language Models from Mixtures to Specialized Domains
Skyler Seto, Pierre Ablin, Anastasiia Filippova +4
Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard t…
The Design Space of Tri-Modal Masked Diffusion Models
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec +21
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal gen…
LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss
Szilvia Ujváry, Louis Béthune, Pierre Ablin +3
Language models have consistently grown to compress more world knowledge into their parameters, but the knowledge that can be pretrained into them is upper-bounded by their paramet…
Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
Bruno Mlodozeniec, Pierre Ablin, Louis Béthune +4
Hyperparameter tuning can dramatically impact training stability and final performance of large-scale models. Recent works on neural network parameterisations, such as P, have e…
Learning Unmasking Policies for Diffusion Language Models
Metod Jazbec, Theo X. Olausson, Louis Béthune +6
Diffusion (Large) Language Models (dLLMs) now match the downstream performance of their autoregressive counterparts on many tasks, while holding the promise of being more efficient…
AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
Jinghang Shi, Xiaoyu Tang, Yang Huang +4
Astronomical image interpretation presents a significant challenge for applying multimodal large language models (MLLMs) to specialized scientific tasks. Existing benchmarks focus…
Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining
Thiziri Nait Saada, Louis Bethune, Michal Klein +3
Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quali…
The Geometries of Truth Are Orthogonal Across Tasks
Waiss Azizian, Michael Kirchhof, Eugene Ndiaye +4
Large Language Models (LLMs) have demonstrated impressive generalization capabilities across various tasks, but their claim to practical relevance is still mired by concerns on the…
Scaling Laws for Optimal Data Mixtures
Mustafa Shukor, Louis Bethune, Dan Busbridge +4
Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance…
Follow the Energy, Find the Path: Riemannian Metrics from Energy-Based Models
Louis Béthune, David Vigouroux, Yilun Du +3
What is the shortest path between two data points lying in a high-dimensional space? While the answer is trivial in Euclidean geometry, it becomes significantly more complex when t…
Deep Sturm--Liouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functions
David Vigouroux, Joseba Dalmau, Louis Béthune +1
Although Artificial Neural Networks (ANNs) have achieved remarkable success across various tasks, they still suffer from limited generalization. We hypothesize that this limitation…
Sample and Map from a Single Convex Potential: Generation using Conjugate Moment Measures
Nina Vesseron, Louis Béthune, Marco Cuturi
The canonical approach in generative modeling is to split model fitting into two blocks: define first how to sample noise (e.g. Gaussian) and choose next what to do with it (e.g. u…
Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlo
James Thornton, Louis Bethune, Ruixiang Zhang +3
Diffusion models may be formulated as a time-indexed sequence of energy-based models, where the score corresponds to the negative gradient of an energy function. As opposed to lear…
Multivariate Conformal Prediction using Optimal Transport
Michal Klein, Louis Bethune, Eugene Ndiaye +1
Conformal prediction (CP) quantifies the uncertainty of machine learning models by constructing sets of plausible outputs. These sets are constructed by leveraging a so-called conf…
Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Louis Bethune, David Grangier, Dan Busbridge +3
A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from…
Multimodal Autoregressive Pre-training of Large Vision Encoders
Enrico Fini, Mustafa Shukor, Xiujun Li +13
We introduce a novel method for pre-training of large-scale vision encoders. Building on recent advancements in autoregressive pre-training of vision models, we extend this framewo…
Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Michael Kirchhof, James Thornton, Louis Béthune +3
The adoption of text-to-image diffusion models raises concerns over reliability, drawing scrutiny under the lens of various metrics like calibration, fairness, or compute efficienc…
Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions
Yu-Guan Hsieh, Cheng-Yu Hsieh, Shih-Ying Yeh +7
Humans describe complex scenes with compositionality, using simple text descriptions enriched with links and relationships. While vision-language research has aimed to develop mode…
TaCo: Targeted Concept Erasure Prevents Non-Linear Classifiers From Detecting Protected Attributes
Fanny Jourdan, Louis Béthune, Agustin Picard +2
Ensuring fairness in NLP models is crucial, as they often encode sensitive attributes like gender and ethnicity, leading to biased outcomes. Current concept erasure methods attempt…