193 citations · 348 across the 28 of their papers we have counts for
5 papers · 1 filter
Bootstrap Your Own Variance
Polina Turishcheva, Jason Ramapuram, Sinead Williamson +3
Understanding model uncertainty is important for many applications. We propose Bootstrap Your Own Variance (BYOV), combining Bootstrap Your Own Latent (BYOL), a negative-free Self-…
How to Scale Your EMA
Dan Busbridge, Jason Ramapuram, Pierre Ablin +4
Preserving training dynamics across batch sizes is an important tool for practical machine learning as it enables the trade-off between batch size and wall-clock time. This trade-o…
The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning
Borja Rodríguez-Gálvez, Arno Blaas, Pau Rodríguez +5
The mechanisms behind the success of multi-view self-supervised learning (MVSSL) are not yet fully understood. Contrastive MVSSL methods have been studied through the lens of InfoN…
DUET: 2D Structured and Approximately Equivariant Representations
Xavier Suau, Federico Danieli, T. Anderson Keller +5
Multiview Self-Supervised Learning (MSSL) is based on learning invariances with respect to a set of input transformations. However, invariance partially or totally removes transfor…
Stabilizing Transformer Training by Preventing Attention Entropy Collapse
Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin +5
Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In…