4 citations · 4 across the 7 of their papers we have counts for
10 papers · 1 filter
The Design Space of Tri-Modal Masked Diffusion Models
Louis Bethune, Victor Turrisi, Bruno Kacper Mlodozeniec +21
Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and fine-tuning a base unimodal model for bimodal gen…
Distillation Scaling Laws
Dan Busbridge, Amitis Shidani, Floris Weers +3
We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate…
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
Jason Ramapuram, Federico Danieli, Eeshan Dhekane +8
Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are t…
Poly-View Contrastive Learning
Amitis Shidani, Devon Hjelm, Jason Ramapuram +3
Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investig…
Bootstrap Your Own Variance
Polina Turishcheva, Jason Ramapuram, Sinead Williamson +3
Understanding model uncertainty is important for many applications. We propose Bootstrap Your Own Variance (BYOV), combining Bootstrap Your Own Latent (BYOL), a negative-free Self-…
Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer
Andrius Ovsianas, Jason Ramapuram, Dan Busbridge +2
Self-supervised representation learning (SSL) methods provide an effective label-free initial condition for fine-tuning downstream tasks. However, in numerous realistic scenarios,…