Bootstrap your own latent: A new approach to self-supervised Learning

Abstract

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.

References in corpus (33)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Continuous control with deep reinforcement learning
• A Simple Framework for Contrastive Learning of Visual Representations
• On the difficulty of training Recurrent Neural Networks
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Improved Baselines with Momentum Contrastive Learning
• Wide Residual Networks
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Semi-Supervised Learning with Deep Generative Models
• Unsupervised Representation Learning by Predicting Image Rotations
• Temporal Ensembling for Semi-Supervised Learning
• Fine-Grained Visual Classification of Aircraft
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Adversarial Feature Learning
• Adversarially Learned Inference
• What Makes for Good Views for Contrastive Learning?
• Semi-Supervised Learning with Ladder Networks
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Prototypical Contrastive Learning of Unsupervised Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Learning with Pseudo-Ensembles
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Deep Reinforcement Learning and the Deadly Triad
• MaxUp: A Simple Way to Improve Generalization of Neural Network Training

Cited by in corpus (456)

• Learning Transferable Visual Models From Natural Language Supervision
• Transformers in Vision: A Survey
• On the Opportunities and Risks of Foundation Models
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• DINOv2: Learning Robust Visual Features without Supervision
• Contrastive Representation Learning: A Framework and Review
• A Comprehensive Survey of Image Augmentation Techniques for Deep Learning
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Graph Self-Supervised Learning: A Survey
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Exploring Simple Siamese Representation Learning
• A Comprehensive Survey on Deep Graph Representation Learning
• Towards artificial general intelligence via a multimodal foundation model
• Contrastive Learning of Medical Visual Representations from Paired Images and Text
• Self-supervised learning methods and applications in medical imaging analysis: A survey
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• Towards Representation Alignment and Uniformity in Collaborative Filtering
• Self-supervised representation learning from 12-lead ECG data
• SPICE: Semantic Pseudo-labeling for Image Clustering
• Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging
• Video Transformers: A Survey
• TCLR: Temporal Contrastive Learning for Video Representation
• Pre-training Molecular Graph Representation with 3D Geometry
• Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Survey
• Twin Contrastive Learning for Online Clustering
• Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structure
• Self-Supervised Learning for Videos: A Survey
• Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data
• DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer
• SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding
• One Million Scenes for Autonomous Driving: ONCE Dataset
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• Learning Representation for Clustering via Prototype Scattering and Positive Sampling
• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
• Recent Progress in Transformer-based Medical Image Analysis
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Search
• Graph Barlow Twins: A self-supervised representation learning framework for graphs
• Disentangling Long and Short-Term Interests for Recommendation
• Cluster-guided Asymmetric Contrastive Learning for Unsupervised Person Re-Identification
• 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
• Self-Supervised Learning with Swin Transformers
• Active label cleaning for improved dataset quality under resource constraints
• Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Series
• Contrast-Phys: Unsupervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast
• Is it Time to Replace CNNs with Transformers for Medical Images?
• Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Self-Supervised Multisensor Change Detection
• What to Hide from Your Students: Attention-Guided Masked Image Modeling
• Learning Disentangled Representations in the Imaging Domain
• Hybrid Dynamic Contrast and Probability Distillation for Unsupervised Person Re-Id
• Learning and Evaluating Representations for Deep One-class Classification
• Vector-quantized Image Modeling with Improved VQGAN
• Self-Supervised Anomaly Detection in Computer Vision and Beyond: A Survey and Outlook
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• HyperNet: Self-Supervised Hyperspectral Spatial-Spectral Feature Understanding Network for Hyperspectral Change Detection
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• Efficient Self-supervised Vision Transformers for Representation Learning
• Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging
• Self-supervised deep convolutional neural network for chest X-ray classification
• CLIP in Medical Imaging: A Survey
• Deep Learning Based Single Sample Per Person Face Recognition: A Survey
• Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis
• SEED: Self-supervised Distillation For Visual Representation
• Exploring scalable medical image encoders beyond text supervision
• Self-EMD: Self-Supervised Object Detection without ImageNet
• Decoupling Representation Learning from Reinforcement Learning
• Self-supervised Learning is More Robust to Dataset Imbalance
• Machine Translation from Signed to Spoken Languages: State of the Art and Challenges
• BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations
• Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• Self-Supervised Pre-Training with Contrastive and Masked Autoencoder Methods for Dealing with Small Datasets in Deep Learning for Medical Imaging
• Aligning Pretraining for Detection via Object-Level Contrastive Learning
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
• i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
• Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection
• MixCo: Mix-up Contrastive Learning for Visual Representation
• APANet: Adaptive Prototypes Alignment Network for Few-Shot Semantic Segmentation
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• A Generic Fundus Image Enhancement Network Boosted by Frequency Self-supervised Representation Learning
• Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Rectifying the Shortcut Learning of Background for Few-Shot Learning
• Self-supervised Audiovisual Representation Learning for Remote Sensing Data
• Whitening for Self-Supervised Representation Learning
• Learning Invariance from Generated Variance for Unsupervised Person Re-identification
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Reinforcement Learning with Prototypical Representations
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training
• Representation Learning via Invariant Causal Mechanisms
• Elephants and Algorithms: A Review of the Current and Future Role of AI in Elephant Monitoring
• Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation
• Contrastive Self-supervised Neural Architecture Search
• Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation
• Contrastive Learning with Stronger Augmentations
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
• Semi-supervised cross-lingual speech emotion recognition
• MoPro: Webly Supervised Learning with Momentum Prototypes
• Structure-aware Visualization Retrieval
• Understanding Self-supervised Learning with Dual Deep Networks
• SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving
• Bridging the Gap Between Patient-specific and Patient-independent Seizure Prediction via Knowledge Distillation
• Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
• VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
• UP-DETR: Unsupervised Pre-training for Object Detection with Transformers
• Contrastive Learning of Musical Representations
• Do Different Tracking Tasks Require Different Appearance Models?
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Cross-Camera Feature Prediction for Intra-Camera Supervised Person Re-identification across Distant Scenes
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• SelfFed: Self-Supervised Federated Learning for Data Heterogeneity and Label Scarcity in Medical Images
• Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence
• CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients
• A Study of the Generalizability of Self-Supervised Representations
• Information Maximization Clustering via Multi-View Self-Labelling
• Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition
• Improving COVID-19 CT Classification of CNNs by Learning Parameter-Efficient Representation
• Unbiased Teacher for Semi-Supervised Object Detection
• Exploring Cross-Domain Pretrained Model for Hyperspectral Image Classification
• Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• ASCON: Anatomy-aware Supervised Contrastive Learning Framework for Low-dose CT Denoising
• A method to challenge symmetries in data with self-supervised learning
• Deep Active Learning for Computer Vision: Past and Future
• ATST: Audio Representation Learning with Teacher-Student Transformer
• Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining
• iART: A Search Engine for Art-Historical Images to Support Research in the Humanities
• Mastering Atari with Discrete World Models
• Self-Supervised Training Enhances Online Continual Learning
• Self-supervision for medical image classification: state-of-the-art performance with ~100 labeled training samples per class
• A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis
• Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization
• Learning Data Association for Multi-Object Tracking using Only Coordinates
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• Hybrid Discriminative-Generative Training via Contrastive Learning
• Towards the Generalization of Contrastive Self-Supervised Learning
• Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science
• OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
• Adaptive Soft Contrastive Learning
• CLDG: Contrastive Learning on Dynamic Graphs
• Relational Self-Supervised Learning on Graphs
• Multi-Scale Hybrid Vision Transformer for Learning Gastric Histology: AI-Based Decision Support System for Gastric Cancer Treatment
• Dual Contrastive Learning for Spatio-temporal Representation
• MambaMIM: Pre-training Mamba with State Space Token Interpolation and its Application to Medical Image Segmentation
• Uncovering the Over-smoothing Challenge in Image Super-Resolution: Entropy-based Quantification and Contrastive Optimization
• Model-Contrastive Federated Learning
• Contrastive Attention for Automatic Chest X-ray Report Generation
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Self-Knowledge Distillation based Self-Supervised Learning for Covid-19 Detection from Chest X-Ray Images
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• BenthicNet: A global compilation of seafloor images for deep learning applications
• Multi-queue Momentum Contrast for Microvideo-Product Retrieval
• Contrastive Learning with Positive-Negative Frame Mask for Music Representation
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning
• Absolute Wrong Makes Better: Boosting Weakly Supervised Object Detection via Negative Deterministic Information
• Self-supervised Pre-training with Hard Examples Improves Visual Representations
• Replication: Contrastive Learning and Data Augmentation in Traffic Classification Using a Flowpic Input Representation
• FoPro-KD: Fourier Prompted Effective Knowledge Distillation for Long-Tailed Medical Image Recognition
• Clinically Labeled Contrastive Learning for OCT Biomarker Classification
• Improving Weakly Supervised Temporal Action Localization by Exploiting Multi-resolution Information in Temporal Domain
• The History of Speech Recognition to the Year 2030
• Generalization in Reinforcement Learning by Soft Data Augmentation
• Targeted Supervised Contrastive Learning for Long-Tailed Recognition
• Joint Learning of Localized Representations from Medical Images and Reports
• TriBYOL: Triplet BYOL for Self-Supervised Representation Learning
• Large-Scale Hyperspectral Image Clustering Using Contrastive Learning
• A Novel Collaborative Self-Supervised Learning Method for Radiomic Data
• NewsEmbed: Modeling News through Pre-trained Document Representations
• Contrastive Learning of Person-independent Representations for Facial Action Unit Detection
• Self-supervised learning unveils change in urban housing from street-level images
• MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption
• Multi-Scale and Multi-Layer Contrastive Learning for Domain Generalization
• Self-Supervised Learning for Medical Image Data with Anatomy-Oriented Imaging Planes
• Active Self-Semi-Supervised Learning for Few Labeled Samples
• A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
• Jointly spatial-temporal representation learning for individual trajectories
• Few-Shot Segmentation with Global and Local Contrastive Learning
• Large-scale graph representation learning with very deep GNNs and self-supervision
• Self-Damaging Contrastive Learning
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records
• Self-supervised Image Clustering from Multiple Incomplete Views via Constrastive Complementary Generation
• FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
• Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding
• Self-supervised Remote Sensing Images Change Detection at Pixel-level
• Multiple Domain Experts Collaborative Learning: Multi-Source Domain Generalization For Person Re-Identification
• Contrastive Learning for Cold-Start Recommendation
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• False Detection (Positives and Negatives) in Object Detection
• Transfer Learning Application of Self-supervised Learning in ARPES
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• CLDA: Contrastive Learning for Semi-Supervised Domain Adaptation
• Continual Contrastive Learning for Image Classification
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• SelfDocSeg: A Self-Supervised vision-based Approach towards Document Segmentation
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• RadBERT-CL: Factually-Aware Contrastive Learning For Radiology Report Classification
• Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
• Self-supervised Learning from a Multi-view Perspective
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
• Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Unifying Local and Global Multimodal Features for Place Recognition in Aliased and Low-Texture Environments
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Towards Good Practices in Self-supervised Representation Learning
• Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
• Spatially Consistent Representation Learning
• Rethinking Self-Supervised Learning: Small is Beautiful
• FedSiam: Towards Adaptive Federated Semi-Supervised Learning
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• Temperature as Uncertainty in Contrastive Learning
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• Multi-Dataset Benchmarks for Masked Identification using Contrastive Representation Learning
• Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
• Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning
• Self-supervised Graph Neural Networks without explicit negative sampling
• Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learning
• Counterfactual contrastive learning: robust representations via causal image synthesis
• Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment
• Temporal Contrastive Graph Learning for Video Action Recognition and Retrieval
• Interpretable agent communication from scratch (with a generic visual processor emerging on the side)
• SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning
• DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learning
• PointSmile: Point Self-supervised Learning via Curriculum Mutual Information
• RegionCL: Can Simple Region Swapping Contribute to Contrastive Learning?
• Improving Transferability of Representations via Augmentation-Aware Self-Supervision
• Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting
• Training GANs with Stronger Augmentations via Contrastive Discriminator
• In-N-Out Generative Learning for Dense Unsupervised Video Segmentation
• MixSiam: A Mixture-based Approach to Self-supervised Representation Learning
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data
• Exploiting Both Domain-specific and Invariant Knowledge via a Win-win Transformer for Unsupervised Domain Adaptation
• Pseudolabel guided pixels contrast for domain adaptive semantic segmentation
• Local Contrastive Feature learning for Tabular Data
• Combining Self-Supervised and Supervised Learning with Noisy Labels
• OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
• Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL
• Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders
• A Generic Self-Supervised Framework of Learning Invariant Discriminative Features
• Self-Supervised Learning for Text Recognition: A Critical Survey
• Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
• Augmentation-aware Self-supervised Learning with Conditioned Projector
• MUSE: Music Recommender System with Shuffle Play Recommendation Enhancement
• Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
• Self supervised learning improves dMMR/MSI detection from histology slides across multiple cancers
• Self-Supervision, Remote Sensing and Abstraction: Representation Learning Across 3 Million Locations
• Revisiting consistency for semi-supervised semantic segmentation
• Transformers for One-Shot Visual Imitation
• Self-Supervised Learning by Estimating Twin Class Distributions
• Scalable Self-Supervised Representation Learning from Spatiotemporal Motion Trajectories for Multimodal Computer Vision
• Simple Contrastive Representation Adversarial Learning for NLP Tasks
• Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning
• DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspective
• Emo-DNA: Emotion Decoupling and Alignment Learning for Cross-Corpus Speech Emotion Recognition
• What Is Considered Complete for Visual Recognition?
• Assessing the Generalizability of Deep Neural Networks-Based Models for Black Skin Lesions
• Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding
• Self-Supervised Siamese Autoencoders
• Complex QA and language models hybrid architectures, Survey
• Joint Data and Feature Augmentation for Self-Supervised Representation Learning on Point Clouds
• Camera Adaptation for Fundus-Image-Based CVD Risk Estimation
• Representation Learning via Consistent Assignment of Views to Clusters
• Reachability Embeddings: Scalable Self-Supervised Representation Learning from Mobility Trajectories for Multimodal Geospatial Computer Vision
• Jo-SRC: A Contrastive Approach for Combating Noisy Labels
• Co-mining: Self-Supervised Learning for Sparsely Annotated Object Detection
• On Initial Pools for Deep Active Learning
• Pseudo-Rehearsal for Continual Learning with Normalizing Flows
• Contrastive Multi-View Textual-Visual Encoding: Towards One Hundred Thousand-Scale One-Shot Logo Identification
• Set Representation Learning with Generalized Sliced-Wasserstein Embeddings
• Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning
• Unsupervised domain adaptation via coarse-to-fine feature alignment method using contrastive learning
• Less is More: Selective Reduction of CT Data for Self-Supervised Pre-Training of Deep Learning Models with Contrastive Learning Improves Downstream Classification Performance
• SimTriplet: Simple Triplet Representation Learning with a Single GPU
• Self-Supervised Ranking for Representation Learning
• Self-supervised Graph Representation Learning via Bootstrapping
• Towards Few-Annotation Learning for Object Detection: Are Transformer-based Models More Efficient ?
• Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation
• Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification
• Winning the CVPR'2021 Kinetics-GEBD Challenge: Contrastive Learning Approach
• Evaluating the Label Efficiency of Contrastive Self-Supervised Learning for Multi-Resolution Satellite Imagery
• CUCL: Codebook for Unsupervised Continual Learning
• Representation learning for improved interpretability and classification accuracy of clinical factors from EEG
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• Unsupervised Pre-training for Person Re-identification
• Learning Image Representations for Content Based Image Retrieval of Radiotherapy Treatment Plans
• Self-Supervised Pre-Training Boosts Semantic Scene Segmentation on LiDAR Data
• Bag of Tricks and A Strong baseline for Image Copy Detection
• Contrastive Neural Processes for Self-Supervised Learning
• Center-wise Local Image Mixture For Contrastive Representation Learning
• MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
• Self-Supervised Representation Learning for Nerve Fiber Distribution Patterns in 3D-PLI
• Deep Contrastive Graph Representation via Adaptive Homotopy Learning
• Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• A Simple Framework for Uncertainty in Contrastive Learning
• Toward High Quality Facial Representation Learning
• Machine Learning for Automated Mitral Regurgitation Detection from Cardiac Imaging
• On Feature Decorrelation in Self-Supervised Learning
• Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
• Contrastive Self-Supervised Learning for Spatio-Temporal Analysis of Lung Ultrasound Videos
• Evolutionary Augmentation Policy Optimization for Self-supervised Learning
• The Power of Contrast for Feature Learning: A Theoretical Analysis
• Induction Network: Audio-Visual Modality Gap-Bridging for Self-Supervised Sound Source Localization
• Simple Distillation Baselines for Improving Small Self-supervised Models
• Whole Heart 3D+T Representation Learning Through Sparse 2D Cardiac MR Images
• Grid Jigsaw Representation with CLIP: A New Perspective on Image Clustering
• Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck
• Scale-Equivariant Imaging: Self-Supervised Learning for Image Super-Resolution and Deblurring
• Revisiting Deep Generalized Canonical Correlation Analysis
• Self-Supervised Learning for Fine-Grained Visual Categorization
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• Unleashing the Potential of Unsupervised Pre-Training with Intra-Identity Regularization for Person Re-Identification
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• A Mathematical Framework for Learning Probability Distributions
• Hierarchical Self-Supervised Learning for Medical Image Segmentation Based on Multi-Domain Data Aggregation
• Multi-Level Graph Contrastive Learning
• Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning
• Geographical Knowledge-driven Representation Learning for Remote Sensing Images
• Multi-Sample based Contrastive Loss for Top-k Recommendation
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration
• Cluster Analysis with Deep Embeddings and Contrastive Learning
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• Morphological Classification of Radio Galaxies using Semi-Supervised Group Equivariant CNNs
• Motion-aware Contrastive Video Representation Learning via Foreground-background Merging
• Are Fewer Labels Possible for Few-shot Learning?
• Self-supervised learning via inter-modal reconstruction and feature projection networks for label-efficient 3D-to-2D segmentation
• Meta Clustering Learning for Large-scale Unsupervised Person Re-identification
• Multi-source Few-shot Domain Adaptation
• TINC: Temporally Informed Non-Contrastive Learning for Disease Progression Modeling in Retinal OCT Volumes
• Shot in the Dark: Few-Shot Learning with No Base-Class Labels
• On The Effect of Auxiliary Tasks on Representation Dynamics
• Self-supervised Scene Text Segmentation with Object-centric Layered Representations Augmented by Text Regions
• Addressing Feature Suppression in Unsupervised Visual Representations
• Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognition
• Contrastive Learning for Self-Supervised Pre-Training of Point Cloud Segmentation Networks With Image Data
• Pretrained Encoders are All You Need
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• Ranking Neural Checkpoints
• Reducing Annotation Need in Self-Explanatory Models for Lung Nodule Diagnosis
• Long-Short Temporal Contrastive Learning of Video Transformers
• A Unified Mixture-View Framework for Unsupervised Representation Learning
• Sensor Data Augmentation by Resampling for Contrastive Learning in Human Activity Recognition
• Tasting the cake: evaluating self-supervised generalization on out-of-distribution multimodal MRI data
• Noise-Resistant Deep Metric Learning with Probabilistic Instance Filtering
• Self-supervised Change Detection in Multi-view Remote Sensing Images
• Disentangled Contrastive Learning for Learning Robust Textual Representations
• D${}^2$LV: A Data-Driven and Local-Verification Approach for Image Copy Detection
• Self-Supervised Transformer Architecture for Change Detection in Radio Access Networks
• Aggregative Self-Supervised Feature Learning from a Limited Sample
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Leveraging Imperfect Restoration for Data Availability Attack
• Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs
• Temporal Knowledge Consistency for Unsupervised Visual Representation Learning
• Inter-intra Variant Dual Representations forSelf-supervised Video Recognition
• CoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised Learning
• Trainable Class Prototypes for Few-Shot Learning
• The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video
• PyTorchVideo: A Deep Learning Library for Video Understanding
• Learning Representations for Pixel-based Control: What Matters and Why?
• Contrastive Learning with Temporal Correlated Medical Images: A Case Study using Lung Segmentation in Chest X-Rays
• Self-Distilled Self-Supervised Representation Learning
• 10 Security and Privacy Problems in Large Foundation Models
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• On the robustness of self-supervised representations for multi-view object classification
• Multi-dataset Pretraining: A Unified Model for Semantic Segmentation
• Prune Your Model Before Distill It
• ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
• Exploring Set Similarity for Dense Self-supervised Representation Learning
• WixUp: A General Data Augmentation Framework for Wireless Perception in Tracking of Humans
• LaFAM: Unsupervised Feature Attribution with Label-free Activation Maps
• Impact of facial landmark localization on facial expression recognition
• Contrastive Learning and Self-Training for Unsupervised Domain Adaptation in Semantic Segmentation
• Deciphering galaxy images using machine vision -- Combining variational autoencoder and principal component analysis for feature extraction
• Parametric Contrastive Learning
• Distilling Stereo Networks for Performant and Efficient Leaner Networks
• CLAWS: Contrastive Learning with hard Attention and Weak Supervision
• Enabling the Network to Surf the Internet
• Domain-Aware Augmentations for Unsupervised Online General Continual Learning
• UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning
• Dynamic Slimmable Network
• Multi-level Knowledge Distillation via Knowledge Alignment and Correlation
• Complementary Relation Contrastive Distillation
• Unsupervised Person Re-identification via Simultaneous Clustering and Consistency Learning
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Brittle Features May Help Anomaly Detection
• Vehicle Trajectory Prediction by Transfer Learning of Semi-Supervised Models
• Spherical Sliced-Wasserstein
• Video Contrastive Learning with Global Context
• Graphs for deep learning representations
• Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners
• Label-free Monitoring of Self-Supervised Learning Progress
• Learning Metrics from Mean Teacher: A Supervised Learning Method for Improving the Generalization of Speaker Verification System
• Bootstrapped Meta-Learning
• PatchGame: Learning to Signal Mid-level Patches in Referential Games
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Improve Unsupervised Pretraining for Few-label Transfer
• MIC: Model-agnostic Integrated Cross-channel Recommenders
• Domain-Agnostic Clustering with Self-Distillation
• SVMax: A Feature Embedding Regularizer
• Self-Supervised Visual Representation Learning Using Lightweight Architectures
• Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation
• Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE
• Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast
• GenURL: A General Framework for Unsupervised Representation Learning
• RUEL: Retrieval-Augmented User Representation with Edge Browser Logs for Sequential Recommendation
• Self-Supervised Learning for Large-Scale Unsupervised Image Clustering
• CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations
• Equivariant Contrastive Learning
• ScaleNet: An Unsupervised Representation Learning Method for Limited Information
• Contrastive Representations for Label Noise Require Fine-Tuning
• Homography augumented momentum constrastive learning for SAR image retrieval
• Voxel-wise Cross-Volume Representation Learning for 3D Neuron Reconstruction
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations
• A Survey on Green Deep Learning
• Evaluating Contrastive Learning on Wearable Timeseries for Downstream Clinical Outcomes
• SERE: Exploring Feature Self-relation for Self-supervised Transformer
• Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection
• Semi-Supervised Learning with Taxonomic Labels
• Using Fictitious Class Representations to Boost Discriminative Zero-Shot Learners
• MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning
• UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection
• Stochastic Contrastive Learning
• Self-supervised similarity search for large scientific datasets
• Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation
• SelfGait: A Spatiotemporal Representation Learning Method for Self-supervised Gait Recognition
• Topo2vec: Topography Embedding Using the Fractal Effect
• Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models
• Semi-supervised Learning for Dense Object Detection in Retail Scenes
• A deep cut into Split Federated Self-supervised Learning
• Sparsity-Probe: Analysis tool for Deep Learning Models
• Visual Probing: Cognitive Framework for Explaining Self-Supervised Image Representations
• Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition
• Three Pillars improving Vision Foundation Model Distillation for Lidar
• Langevin Cooling for Domain Translation
• Knowledge accumulating: The general pattern of learning
• Pretext Tasks selection for multitask self-supervised speech representation learning
• On-target Adaptation
• Data-Driven Self-Supervised Graph Representation Learning
• SLADE: A Self-Training Framework For Distance Metric Learning
• FROST: Faster and more Robust One-shot Semi-supervised Training
• Super-Selfish: Self-Supervised Learning on Images with PyTorch
• Color Variants Identification in Fashion e-commerce via Contrastive Self-Supervised Representation Learning
• Learning Representations on the Unit Sphere: Investigating Angular Gaussian and von Mises-Fisher Distributions for Online Continual Learning
• Explaining Representation by Mutual Information
• Deep Bregman Divergence for Contrastive Learning of Visual Representations