128 citations · 179 across the 23 of their papers we have counts for
23 papers
How PARTs assemble into wholes: Learning the relative composition of images
Melika Ayoughi, Samira Abnar, Chen Huang +10
The composition of objects and their parts, along with object-object positional relationships, provides a rich source of information for representation learning. Hence, spatial-awa…
Distillation Scaling Laws
Dan Busbridge, Amitis Shidani, Floris Weers +3
We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate…
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
Samira Abnar, Harshay Shah, Dan Busbridge +3
Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily define…
Poly-View Contrastive Learning
Amitis Shidani, Devon Hjelm, Jason Ramapuram +3
Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investig…
Bootstrap Your Own Variance
Polina Turishcheva, Jason Ramapuram, Sinead Williamson +3
Understanding model uncertainty is important for many applications. We propose Bootstrap Your Own Variance (BYOV), combining Bootstrap Your Own Latent (BYOL), a negative-free Self-…
REALM: Robust Entropy Adaptive Loss Minimization for Improved Single-Sample Test-Time Adaptation
Skyler Seto, Barry-John Theobald, Federico Danieli +2
Fully-test-time adaptation (F-TTA) can mitigate performance loss due to distribution shifts between train and test data (1) without access to the training data, and (2) without kno…
How to Scale Your EMA
Dan Busbridge, Jason Ramapuram, Pierre Ablin +4
Preserving training dynamics across batch sizes is an important tool for practical machine learning as it enables the trade-off between batch size and wall-clock time. This trade-o…
The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning
Borja Rodríguez-Gálvez, Arno Blaas, Pau Rodríguez +5
The mechanisms behind the success of multi-view self-supervised learning (MVSSL) are not yet fully understood. Contrastive MVSSL methods have been studied through the lens of InfoN…
DUET: 2D Structured and Approximately Equivariant Representations
Xavier Suau, Federico Danieli, T. Anderson Keller +5
Multiview Self-Supervised Learning (MSSL) is based on learning invariances with respect to a set of input transformations. However, invariance partially or totally removes transfor…
Stabilizing Transformer Training by Preventing Attention Entropy Collapse
Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin +5
Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In…
Elastic Weight Consolidation Improves the Robustness of Self-Supervised Learning Methods under Transfer
Andrius Ovsianas, Jason Ramapuram, Dan Busbridge +2
Self-supervised representation learning (SSL) methods provide an effective label-free initial condition for fine-tuning downstream tasks. However, in numerous realistic scenarios,…
Position Prediction as an Effective Pretraining Strategy
Shuangfei Zhai, Navdeep Jaitly, Jason Ramapuram +7
Transformers have gained increasing popularity in a wide range of applications, including Natural Language Processing (NLP), Computer Vision and Speech Recognition, because of thei…
Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?
Tom George Grigg, Dan Busbridge, Jason Ramapuram +1
Despite the success of a number of recent techniques for visual self-supervised deep learning, there has been limited investigation into the representations that are ultimately lea…
Stochastic Contrastive Learning
Jason Ramapuram, Dan Busbridge, Xavier Suau +1
While state-of-the-art contrastive Self-Supervised Learning (SSL) models produce results competitive with their supervised counterparts, they lack the ability to infer latent varia…
Evaluating the fairness of fine-tuning strategies in self-supervised learning
Jason Ramapuram, Dan Busbridge, Russ Webb
In this work we examine how fine-tuning impacts the fairness of contrastive Self-Supervised Learning (SSL) models. Our findings indicate that Batch Normalization (BN) statistics pl…
Neural Temporal Point Processes For Modelling Electronic Health Records
Joseph Enguehard, Dan Busbridge, Adam Bozson +2
The modelling of Electronic Health Records (EHRs) has the potential to drive more efficient allocation of healthcare resources, enabling early intervention strategies and advancing…
Learning medical triage from clinicians using Deep Q-Learning
Albert Buchard, Baptiste Bouvier, Giulia Prando +10
Medical Triage is of paramount importance to healthcare systems, allowing for the correct orientation of patients and allocation of the necessary resources to treat them adequately…
Correlations between Word Vector Sets
Vitalii Zhelezniak, April Shen, Daniel Busbridge +2
Similarity measures based purely on word embeddings are comfortably competing with much more sophisticated deep learning and expert-engineered systems on unsupervised semantic text…
Neural Language Priors
Joseph Enguehard, Dan Busbridge, Vitalii Zhelezniak +1
The choice of sentence encoder architecture reflects assumptions about how a sentence's meaning is composed from its constituent words. We examine the contribution of these archite…
Relational Graph Attention Networks
Dan Busbridge, Dane Sherburn, Pietro Cavallo +1
We investigate Relational Graph Attention Networks, a class of models that extends non-relational graph attention mechanisms to incorporate relational information, opening up these…
Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks
Vitalii Zhelezniak, Dan Busbridge, April Shen +2
Experimental evidence indicates that simple models outperform complex deep networks on many unsupervised similarity tasks. We provide a simple yet rigorous explanation for this beh…
Constrained Dirac gluino mediation
Daniel Busbridge
We perform a comparison study of the Constrained Minimal Supersymmetric Standard Model and Constrained General Gauge Mediation with and without a heavy Dirac gluino. These extremel…
Mapping Dirac gaugino masses
Steven Abel, Daniel Busbridge
We investigate the mapping of Dirac gaugino masses through regions of strong coupling, focussing on SQCD with an adjoint. These models have a well-known Kutasov duality, under whic…