2k citations · 4.8k across the 35 of their papers we have counts for
38 papers
Stochastic positional embeddings improve masked image modeling
Amir Bar, Florian Bordes, Assaf Shocher +6
Masked Image Modeling (MIM) is a promising self-supervised learning approach that enables learning from unlabeled images. Despite its recent success, learning good representations…
A surprisingly simple technique to control the pretraining bias for better transfer: Expand or Narrow your representation
Florian Bordes, Samuel Lavoie, Randall Balestriero +2
Self-Supervised Learning (SSL) models rely on a pretext task to learn representations. Because this pretext task differs from the downstream tasks used to evaluate the performance…
DINOv2: Learning Robust Visual Features without Supervision
Maxime Oquab, Timothée Darcet, Théo Moutakanni +23
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. Thes…
A Simple Recipe for Competitive Low-compute Self supervised Vision Models
Quentin Duval, Ishan Misra, Nicolas Ballas
Self-supervised methods in vision have been mostly focused on large architectures as they seem to suffer from a significant performance drop for smaller architectures. In this pape…
Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
Mahmoud Assran, Quentin Duval, Ishan Misra +5
This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations. We introduce the Image-based Joint-Embedd…
Uniform Masking Prevails in Vision-Language Pretraining
Siddharth Verma, Yuchen Lu, Rui Hou +4
Masked Language Modeling (MLM) has proven to be an essential component of Vision-Language (VL) pretraining. To implement MLM, the researcher must make two design choices: the maski…
ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations
Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero +7
Deep learning vision systems are widely deployed across applications where reliability is critical. However, even today's best models can fail to recognize an object when its pose,…
Neural Attentive Circuits
Nasim Rahaman, Martin Weiss, Francesco Locatello +5
Recent work has seen the development of general purpose neural architectures that can be trained to perform tasks across diverse data modalities. General purpose models typically m…
The Hidden Uniform Cluster Prior in Self-Supervised Learning
Mahmoud Assran, Randall Balestriero, Quentin Duval +6
A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e.g., SimCLR, VICReg, SwAV, MSN). We show tha…
Cascaded Video Generation for Videos In-the-Wild
Lluis Castrejon, Nicolas Ballas, Aaron Courville
Videos can be created by first outlining a global view of the scene and then adding local details. Inspired by this idea we propose a cascaded model for video generation which foll…
Masked Siamese Networks for Label-Efficient Learning
Mahmoud Assran, Mathilde Caron, Ishan Misra +6
We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containi…
BARACK: Partially Supervised Group Robustness With Guarantees
Nimit S. Sohoni, Maziar Sanjabi, Nicolas Ballas +4
While neural networks have shown remarkable success on classification tasks in terms of average-case performance, they often fail to perform well on certain groups of the data. Suc…
Trade-offs of Local SGD at Scale: An Empirical Study
Jose Javier Gonzalez Ortiz, Jonathan Frankle, Mike Rabbat +2
As datasets and models become increasingly large, distributed training has become a necessary component to allow deep neural networks to train in reasonable amounts of time. Howeve…
Hierarchical Video Generation for Complex Data
Lluis Castrejon, Nicolas Ballas, Aaron Courville
Videos can often be created by first outlining a global description of the scene and then adding local details. Inspired by this we propose a hierarchical model for video generatio…
Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples
Mahmoud Assran, Mathilde Caron, Ishan Misra +4
This paper proposes a novel method of learning by predicting view assignments with support samples (PAWS). The method trains a model to minimize a consistency loss, which ensures t…
A Closer Look at Codistillation for Distributed Training
Shagun Sodhani, Olivier Delalleau, Mahmoud Assran +3
Codistillation has been proposed as a mechanism to share knowledge among concurrently trained models by encouraging them to represent the same function through an auxiliary loss. T…
Revisiting Loss Modelling for Unstructured Pruning
César Laurent, Camille Ballas, Thomas George +2
By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. I…
Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
Mahmoud Assran, Nicolas Ballas, Lluis Castrejon +1
We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training.…
SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
Jianyu Wang, Vinayak Tantia, Nicolas Ballas +1
Distributed optimization is essential for training large models on large datasets. Multiple approaches have been proposed to reduce the communication overhead in distributed traini…
Needles in Haystacks: On Classifying Tiny Objects in Large Images
Nick Pawlowski, Suvrat Bhooshan, Nicolas Ballas +3
In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional N…
Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning
Mahmoud Assran, Joshua Romoff, Nicolas Ballas +2
Multi-simulator training has contributed to the recent success of Deep Reinforcement Learning by stabilizing learning and allowing for higher training throughputs. We propose Gossi…
Improved Conditional VRNNs for Video Prediction
Lluis Castrejon, Nicolas Ballas, Aaron Courville
Predicting future frames for a video sequence is a challenging generative modeling task. Promising approaches include probabilistic latent variable models such as the Variational A…
Stochastic Gradient Push for Distributed Deep Learning
Mahmoud Assran, Nicolas Loizou, Nicolas Ballas +1
Distributed data-parallel algorithms aim to accelerate the training of deep neural networks by parallelizing the computation of large mini-batch gradient updates across multiple no…
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
Stanisław Jastrzębski, Zachary Kenton, Nicolas Ballas +3
Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight…