A Cookbook of Self-Supervised Learning
arXiv:2304.12210
Abstract
Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.
Cited by in corpus (24)
- General Purpose Artificial Intelligence Systems (GPAIS): Properties, Definition, Taxonomy, Societal Implications and Responsible Governance
- Predicting Gradient is Better: Exploring Self-Supervised Learning for SAR ATR with a Joint-Embedding Predictive Architecture
- HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
- Machine Learning for Anomaly Detection in Particle Physics
- AstroCLIP: A Cross-Modal Foundation Model for Galaxies
- Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology
- Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
- BenthicNet: A global compilation of seafloor images for deep learning applications
- Data Compression and Inference in Cosmology with Self-Supervised Machine Learning
- Self-supervised learning unveils change in urban housing from street-level images
- 3DTINC: Time-Equivariant Non-Contrastive Learning for Predicting Disease Progression from Longitudinal OCTs
- Augmentation-aware Self-supervised Learning with Conditioned Projector
- Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders
- Less is More: Selective Reduction of CT Data for Self-Supervised Pre-Training of Deep Learning Models with Contrastive Learning Improves Downstream Classification Performance
- BarcodeBERT: Transformers for Biodiversity Analysis
- Learning Robust Features for Scatter Removal and Reconstruction in Dynamic ICF X-Ray Tomography
- Understanding the limitations of self-supervised learning for tabular anomaly detection
- Unsupervised learning on spontaneous retinal activity leads to efficient neural representation geometry
- MAP-Elites with Transverse Assessment for Multimodal Problems in Creative Domains
- Unsupervised End-to-End Training with a Self-Defined Target
- Self-supervised Benchmark Lottery on ImageNet: Do Marginal Improvements Translate to Improvements on Similar Datasets?
- General Feature Extraction In SAR Target Classification: A Contrastive Learning Approach Across Sensor Types
- Three Pillars improving Vision Foundation Model Distillation for Lidar
- Optimal thresholds and algorithms for a model of multi-modal learning in high dimensions