1.1k citations · 1.4k across the 16 of their papers we have counts for
8 papers · 1 filter
Global Exponential Convergence of Two-Layer Linear Network Training
Stephen Y Zhang, Gabriel Peyré
We prove global exponential (linear) convergence with an explicit rate in the rich scaling for wide two-layer linear networks trained with smooth Polyak-Lojasiewicz predictor losse…
The Hidden Uniform Cluster Prior in Self-Supervised Learning
Mahmoud Assran, Randall Balestriero, Quentin Duval +6
A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e.g., SimCLR, VICReg, SwAV, MSN). We show tha…
Masked Siamese Networks for Label-Efficient Learning
Mahmoud Assran, Mathilde Caron, Ishan Misra +6
We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containi…
A Closer Look at Codistillation for Distributed Training
Shagun Sodhani, Olivier Delalleau, Mahmoud Assran +3
Codistillation has been proposed as a mechanism to share knowledge among concurrently trained models by encouraging them to represent the same function through an auxiliary loss. T…
Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
Mahmoud Assran, Nicolas Ballas, Lluis Castrejon +1
We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training.…
On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings
Mahmoud Assran, Michael Rabbat
We study Nesterov's accelerated gradient method with constant step-size and momentum parameters in the stochastic approximation setting (unbiased gradients with bounded variance) a…
Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
Mahmoud Assran, Joshua Romoff, Nicolas Ballas +2
Multi-simulator training has contributed to the recent success of Deep Reinforcement Learning by stabilizing learning and allowing for higher training throughputs. We propose Gossi…
Stochastic Gradient Push for Distributed Deep Learning
Mahmoud Assran, Nicolas Loizou, Nicolas Ballas +1
Distributed data-parallel algorithms aim to accelerate the training of deep neural networks by parallelizing the computation of large mini-batch gradient updates across multiple no…