3 citations · 5 across the 33 of their papers we have counts for
27 papers · 1 filter
Cartridges++: KV Cache Compression without Off-Context Derailment
Sonia Laguna, Joao Monteiro, Marco Cuturi +2
Serving long documents to a Large Language Model (LLM) repeatedly is expensive: computations grow with context length, and the memory footprint of the key-value (KV) cache balloons…
Brenier Meets Adversarial Training: Optimal Transport Geometry for Robust Learning
Alireza Abdollahpoorrostam, Ehsan Sharifian, Buse Şen +2
Distributionally robust optimization (DRO) provides a principled framework for learning under distribution shift, but its practical use is hindered by the difficulty of evaluating…
Locking Pretrained Weights via Deep Low-Rank Residual Distillation
Keitaro Sakamoto, Pierre Ablin, Federico Danieli +1
The quality of open-weight language models has dramatically improved in recent years. Sharing weights greatly facilitates model adoption by enabling their use across diverse hardwa…
DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
Eleonora Gualdoni, Sonia Laguna, Louis Bethune +3
Multi-domain fine-tuning of large language models requires improving performance on target domains while preserving previous capabilities, such as general knowledge, instruction fo…
Scaling Categorical Flow Maps
Oscar Davis, Anastasiia Filippova, Pierre Ablin +4
Continuous diffusion and flow matching models could represent a powerful alternative to autoregressive approaches for language modelling (LM), as they unlock a host of advantages c…
HyperTransport: Amortized Conditioning of T2I Generative Models
Valentino Maiorca, Eleonora Gualdoni, Xavier Suau +3
As foundation models grow in capability, the ability to efficiently and reliably control their behavior becomes critical. Fine-tuning these models can be costly, and while promptin…
Nectar: Neural Estimation of Cached-Token Attention via Regression
João Monteiro, Michal Klein, Pierre Ablin +1
Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token. For a given context (a book, a manual, a legal corpus)…
Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
Anastasiia Filippova, David Grangier, Marco Cuturi +1
Serving transformer language models with high throughput requires caching Key-Values (KVs) to avoid redundant computation during autoregressive generation. The memory footprint of…
The Coupling Within: Flow Matching via Distilled Normalizing Flows
David Berthelot, Tianrong Chen, Jiatao Gu +6
Flow models have rapidly become the go-to method for training and deploying large-scale generators, owing their success to inference-time flexibility via adjustable integration ste…
Amortized Maximum Inner Product Search with Learned Support Functions
Theo X. Olausson, João Monteiro, Michal Klein +1
Maximum inner product search (MIPS) is a crucial subroutine in machine learning, requiring the identification of a vector taken within a database (the keys) that best aligns with a…
The Design Space of Tri-Modal Masked Diffusion Models
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec +21
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal gen…
Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
Bruno Mlodozeniec, Pierre Ablin, Louis Béthune +4
Hyperparameter tuning can dramatically impact training stability and final performance of large-scale models. Recent works on neural network parameterisations, such as P, have e…
Learning Unmasking Policies for Diffusion Language Models
Metod Jazbec, Theo X. Olausson, Louis Béthune +6
Diffusion (Large) Language Models (dLLMs) now match the downstream performance of their autoregressive counterparts on many tasks, while holding the promise of being more efficient…
Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining
Thiziri Nait Saada, Louis Bethune, Michal Klein +3
Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quali…
Flow Matching with Semidiscrete Couplings
Alireza Mousavi-Hosseini, Stephen Y. Zhang, Michal Klein +1
Flow models parameterized as time-dependent velocity fields can generate data from noise by integrating an ODE. These models are often trained using flow matching, i.e. by sampling…
The Geometries of Truth Are Orthogonal Across Tasks
Waiss Azizian, Michael Kirchhof, Eugene Ndiaye +4
Large Language Models (LLMs) have demonstrated impressive generalization capabilities across various tasks, but their claim to practical relevance is still mired by concerns on the…
On Fitting Flow Models with Large Sinkhorn Couplings
Stephen Zhang, Alireza Mousavi-Hosseini, Michal Klein +1
Flow models transform data gradually from one modality (e.g. noise) onto another (e.g. images). Such models are parameterized by a time-dependent velocity field, trained to fit seg…
Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Louis Bethune, David Grangier, Dan Busbridge +3
A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from…
Leveraging Cardiovascular Simulations for In-Vivo Prediction of Cardiac Biomarkers
Laura Manduchi, Antoine Wehenkel, Jens Behrmann +6
Whole-body hemodynamics simulators, which model blood flow and pressure waveforms as functions of physiological parameters, are now essential tools for studying cardiovascular syst…
Controlling Language and Diffusion Models by Transporting Activations
Pau Rodriguez, Arno Blaas, Michal Klein +4
The increasing capabilities of large generative models and their ever more widespread deployment have raised concerns about their reliability, safety, and potential misuse. To addr…
Simple ReFlow: Improved Techniques for Fast Flow Models
Beomsu Kim, Yu-Guan Hsieh, Michal Klein +4
Diffusion and flow-matching models achieve remarkable generative performance but at the cost of many sampling steps, this slows inference and limits applicability to time-critical…
Disentangled Representation Learning with the Gromov-Monge Gap
Théo Uscidda, Luca Eyring, Karsten Roth +3
Learning disentangled representations from unlabelled data is a fundamental challenge in machine learning. Solving it may unlock other problems, such as generalization, interpretab…
Contrasting Multiple Representations with the Multi-Marginal Matching Gap
Zoe Piran, Michal Klein, James Thornton +1
Learning meaningful representations of complex objects that can be seen through multiple () views or modalities is a core task in machine learning. Existing methods use lo…
A Galois theorem for machine learning: Functions on symmetric matrices and point clouds via lightweight invariant features
Ben Blum-Smith, Ningyuan Huang, Marco Cuturi +1
In this work, we present a mathematical formulation for machine learning of (1) functions on symmetric matrices that are invariant with respect to the action of permutations by con…