Self-supervised Pretraining of Visual Features in the Wild

Abstract

Recently, self-supervised learning methods like MoCo, SimCLR, BYOL and SwAV have reduced the gap with supervised methods. These results have been achieved in a control environment, that is the highly curated ImageNet dataset. However, the premise of self-supervised learning is that it can learn from any random image and from any unbounded dataset. In this work, we explore if self-supervision lives to its expectation by training large models on random, uncurated images with no supervision. Our final SElf-supERvised (SEER) model, a RegNetY with 1.3B parameters trained on 1B random images with 512 GPUs achieves 84.2% top-1 accuracy, surpassing the best self-supervised pretrained model by 1% and confirming that self-supervised learning works in a real world setting. Interestingly, we also observe that self-supervised models are good few-shot learners achieving 77.9% top-1 with access to only 10% of ImageNet. Code: https://github.com/facebookresearch/vissl

References in corpus (13)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Training Deep Nets with Sublinear Memory Cost
• Large Batch Training of Convolutional Networks
• Prototypical Contrastive Learning of Unsupervised Representations
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Fixing the train-test resolution discrepancy: FixEfficientNet

Cited by in corpus (34)

• DINOv2: Learning Robust Visual Features without Supervision
• Compute Trends Across Three Eras of Machine Learning
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• Masked Autoencoders Are Scalable Vision Learners
• Swin Transformer V2: Scaling Up Capacity and Resolution
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• Efficient Self-supervised Vision Transformers for Representation Learning
• Self-supervised Learning is More Robust to Dataset Imbalance
• Aligning Pretraining for Detection via Object-Level Contrastive Learning
• Exploring the Limits of Large Scale Pre-training
• A Study of the Generalizability of Self-Supervised Representations
• Self-supervised visual learning in the low-data regime: a comparative evaluation
• Object-aware Contrastive Learning for Debiased Scene Representation
• Anatomically aware dual-hop learning for pulmonary embolism detection in CT pulmonary angiograms
• Tracking Passengers and Baggage Items Using Multiple Overhead Cameras at Security Checkpoints
• Tracking Passengers and Baggage Items using Multiple Overhead Cameras at Security Checkpoints
• Terrain-Informed Self-Supervised Learning: Enhancing Building Footprint Extraction from LiDAR Data with Limited Annotations
• INTERN: A New Learning Paradigm Towards General Vision
• Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices
• Unsupervised Object-Level Representation Learning from Scene Images
• What Is Considered Complete for Visual Recognition?
• EBJR: Energy-Based Joint Reasoning for Adaptive Inference
• Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagery
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• Nonequilibrium thermodynamics of self-supervised learning
• Object-Aware Cropping for Self-Supervised Learning
• Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning
• Homography augumented momentum constrastive learning for SAR image retrieval
• Batch Curation for Unsupervised Contrastive Representation Learning
• Learning to See by Looking at Noise
• Inductive biases and Self Supervised Learning in modelling a physical heating system
• Self-supervised similarity search for large scientific datasets