263 citations · 392 across the 53 of their papers we have counts for
9 papers · 1 filter
Dynamic Gradient Alignment for Online Data Mixing
Simin Fan, David Grangier, Pierre Ablin
The composition of training data mixtures is critical for effectively training large language models (LLMs), as it directly impacts their performance on downstream tasks. Our goal…
Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Michael Kirchhof, James Thornton, Louis Béthune +3
The adoption of text-to-image diffusion models raises concerns over reliability, drawing scrutiny under the lens of various metrics like calibration, fairness, or compute efficienc…
Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
David Grangier, Simin Fan, Skyler Seto +1
Specialist language models (LMs) focus on a specific task or domain on which they often outperform generalist LMs of the same size. However, the specialist data needed to pretrain…
The AdEMAMix Optimizer: Better, Faster, Older
Matteo Pagliardini, Pierre Ablin, David Grangier
Momentum based optimizers are central to a wide range of machine learning applications. These typically rely on an Exponential Moving Average (EMA) of gradients, which decays expon…
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
Jason Ramapuram, Federico Danieli, Eeshan Dhekane +8
Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are t…
Optimization without Retraction on the Random Generalized Stiefel Manifold
Simon Vary, Pierre Ablin, Bin Gao +1
Optimization over the set of matrices that satisfy , referred to as the generalized Stiefel manifold, appears in many applications involving sampled covarianc…