15 papers
ORCA: Hunting Compositional Failures in Text-to-Image Diffusion
Arshia Hemmat, Amirhossein Vahidi, Amitis Shidani +4
Text-to-image diffusion models fail predictably on compositional prompts: attributes bind to the wrong objects, spatial relations invert, and multi-object scenes lose count. Recent…
On Trajectory-Aware Training for Masked Diffusion Language Models
Manuel Madeira, Amitis Shidani, Alice Bizeul +6
Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The model is trained on randomly mas…
Limits of Confidence in Diffusion
Russ Webb, Amitis Shidani, Alice Bizeul +1
Discrete diffusion, including remasking and uniform-state samplers, generate a sequence by writing multiple token positions per step, drawing each from a per-position distribution…
Scaling Categorical Flow Maps
Oscar Davis, Anastasiia Filippova, Pierre Ablin +4
Continuous diffusion and flow matching models could represent a powerful alternative to autoregressive approaches for language modelling (LM), as they unlock a host of advantages c…
Scaling Properties of Continuous Diffusion Spoken Language Models
Jason Ramapuram, Eeshan Gunesh Dhekane, Amitis Shidani +6
Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational…
The Design Space of Tri-Modal Masked Diffusion Models
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec +21
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal gen…
Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
Jakub Krajewski, Amitis Shidani, Dan Busbridge +2
While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliabl…
Beyond Real Data: Synthetic Data through the Lens of Regularization
Amitis Shidani, Tyler Farghly, Yang Sun +2
Synthetic data can improve generalization when real data is scarce, but excessive reliance may introduce distributional mismatches that degrade performance. In this paper, we prese…
Bernstein-type dimension-free concentration for self-normalised martingales
Arya Akhavan, Amitis Shidani, Alex Ayoub +1
We introduce a dimension-free Bernstein-type tail inequality for self-normalised martingales, where the normalisation uses the predictable quadratic variation and the radius depend…
Distillation Scaling Laws
Dan Busbridge, Amitis Shidani, Floris Weers +3
We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate…
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
Jason Ramapuram, Federico Danieli, Eeshan Dhekane +8
Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are t…
Poly-View Contrastive Learning
Amitis Shidani, Devon Hjelm, Jason Ramapuram +3
Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investig…
Optimal Regret Bounds for Collaborative Learning in Bandits
Amitis Shidani, Sattar Vakili
We consider regret minimization in a general collaborative multi-agent multi-armed bandit model, in which each agent faces a finite set of arms and may communicate with other agent…
Ranking In Generalized Linear Bandits
Amitis Shidani, George Deligiannidis, Arnaud Doucet
We study the ranking problem in generalized linear bandits. At each time, the learning agent selects an ordered list of items and observes stochastic outcomes. In recommendation sy…
Chained Generalisation Bounds
Eugenio Clerico, Amitis Shidani, George Deligiannidis +1
This work discusses how to derive upper bounds for the expected generalisation error of supervised learning algorithms by means of the chaining technique. By developing a general t…