91 citations · 92 across the 4 of their papers we have counts for
4 papers · 1 filter
QUEST: A robust attention formulation using query-modulated spherical attention
Hariprasath Govindarajan, Per Sidén, Jacob Roll +1
The Transformer model architecture has become one of the most widely used in deep learning and the attention mechanism is at its core. The standard attention formulation uses a sof…
On Partial Prototype Collapse in the DINO Family of Self-Supervised Methods
Hariprasath Govindarajan, Per Sidén, Jacob Roll +1
A prominent self-supervised learning paradigm is to model the representations as clusters, or more generally as a mixture model. Learning to map the data samples to compact represe…
DINO as a von Mises-Fisher mixture model
Hariprasath Govindarajan, Per Sidén, Jacob Roll +1
Self-distillation methods using Siamese networks are popular for self-supervised pre-training. DINO is one such method based on a cross-entropy loss between -dimensional probabi…
Evaluating model calibration in classification
Juozas Vaicenavicius, David Widmann, Carl Andersson +3
Probabilistic classifiers output a probability distribution on target classes rather than just a class prediction. Besides providing a clear separation of prediction and decision m…