51 citations · 51 across the 7 of their papers we have counts for
10 papers
On Trajectory-Aware Training for Masked Diffusion Language Models
Manuel Madeira, Amitis Shidani, Alice Bizeul +6
Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The model is trained on randomly mas…
Limits of Confidence in Diffusion
Russ Webb, Amitis Shidani, Alice Bizeul +1
Discrete diffusion, including remasking and uniform-state samplers, generate a sequence by writing multiple token positions per step, drawing each from a per-position distribution…
Scaling Properties of Continuous Diffusion Spoken Language Models
Jason Ramapuram, Eeshan Gunesh Dhekane, Amitis Shidani +6
Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational…
The Design Space of Tri-Modal Masked Diffusion Models
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec +21
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal gen…
Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
Bruno Mlodozeniec, Pierre Ablin, Louis Béthune +4
Hyperparameter tuning can dramatically impact training stability and final performance of large-scale models. Recent works on neural network parameterisations, such as P, have e…
Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
Jakub Krajewski, Amitis Shidani, Dan Busbridge +2
While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliabl…
Scaling Laws for Optimal Data Mixtures
Mustafa Shukor, Louis Bethune, Dan Busbridge +4
Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance…
Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Louis Bethune, David Grangier, Dan Busbridge +3
A widespread strategy to obtain a language model that performs well on a target domain is to finetune a pretrained model to perform unsupervised next-token prediction on data from…
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
Jason Ramapuram, Federico Danieli, Eeshan Dhekane +8
Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are t…
The Impact of Explanations on Layperson Trust in Artificial Intelligence-Driven Symptom Checker Apps: Experimental Study
Claire Woodcock, Brent Mittelstadt, Dan Busbridge +1
To achieve the promoted benefits of an AI symptom checker, laypeople must trust and subsequently follow its instructions. In AI, explanations are seen as a tool to communicate the…