Barlow Twins: Self-Supervised Learning via Redundancy Reduction

Abstract

Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.

arXiv comment13 pages, 6 figures, to appear at ICML 2021

References in corpus (19)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• The information bottleneck method
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Large Batch Training of Convolutional Networks
• Deep Clustering for Unsupervised Learning of Visual Features
• Exploring Simple Siamese Representation Learning
• Unsupervised Learning by Predicting Noise
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Law of Log Determinant of Sample Covariance Matrix and Optimal Estimation of Differential Entropy for High-Dimensional Gaussian Distributions
• Whitening for Self-Supervised Representation Learning
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Understanding Self-supervised Learning with Dual Deep Networks
• BYOL works even without batch statistics

Cited by in corpus (121)

• Graph Self-Supervised Learning: A Survey
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Towards Representation Alignment and Uniformity in Collaborative Filtering
• Video Transformers: A Survey
• Twin Contrastive Learning for Online Clustering
• Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structure
• Graph Barlow Twins: A self-supervised representation learning framework for graphs
• CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification
• CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Learning Disentangled Representations in the Imaging Domain
• SelfCF: A Simple Framework for Self-supervised Collaborative Filtering
• Understanding Dimensional Collapse in Contrastive Self-supervised Learning
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• 3D Infomax improves GNNs for Molecular Property Prediction
• Large-scale Unsupervised Semantic Segmentation
• Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast
• CLIP in Medical Imaging: A Survey
• DualNet: Continual Learning, Fast and Slow
• Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis
• Aligning Pretraining for Detection via Object-Level Contrastive Learning
• Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Inter-Domain Mixup for Semi-Supervised Domain Adaptation
• Do Different Tracking Tasks Require Different Appearance Models?
• Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence
• A Study of the Generalizability of Self-Supervised Representations
• False: False Negative Samples Aware Contrastive Learning for Semantic Segmentation of High-Resolution Remote Sensing Image
• Self-supervised visual learning in the low-data regime: a comparative evaluation
• Self-Supervised Training Enhances Online Continual Learning
• A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis
• Self-Supervised Learning with Kernel Dependence Maximization
• Towards the Generalization of Contrastive Self-Supervised Learning
• GaitFormer: Learning Gait Representations with Noisy Multi-Task Learning
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• Contrastive Learning with Positive-Negative Frame Mask for Music Representation
• Foundation Models and Transformers for Anomaly Detection: A Survey
• Large-Scale Hyperspectral Image Clustering Using Contrastive Learning
• Self-supervised learning unveils change in urban housing from street-level images
• Learning From Long-Tailed Data With Noisy Labels
• Universum-inspired Supervised Contrastive Learning
• Contrastive Learning of Person-independent Representations for Facial Action Unit Detection
• A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Test time Adaptation through Perturbation Robustness
• Semi-supervised Open-World Object Detection
• Temperature as Uncertainty in Contrastive Learning
• Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learning
• Rethinking Supervised Pre-training for Better Downstream Transferring
• Non-contrastive representation learning for intervals from well logs
• DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning
• A Softmax-free Loss Function Based on Predefined Optimal-distribution of Latent Features for Deep Learning Classifier
• SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives
• Unsupervised Pre-Training for 3D Leaf Instance Segmentation
• MUSE: Music Recommender System with Shuffle Play Recommendation Enhancement
• Self-Supervised Learning for Text Recognition: A Critical Survey
• Local Contrastive Feature learning for Tabular Data
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• INoD: Injected Noise Discriminator for Self-Supervised Representation Learning in Agricultural Fields
• Self-Supervised Learning by Estimating Twin Class Distributions
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• TURBO: The Swiss Knife of Auto-Encoders
• Contrastive Multi-View Textual-Visual Encoding: Towards One Hundred Thousand-Scale One-Shot Logo Identification
• CUCL: Codebook for Unsupervised Continual Learning
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
• Contrastive Self-Supervised Learning for Spatio-Temporal Analysis of Lung Ultrasound Videos
• Self-Supervised Representation Learning for Nerve Fiber Distribution Patterns in 3D-PLI
• Self-Supervised Pre-Training Boosts Semantic Scene Segmentation on LiDAR Data
• MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
• Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
• "Are you sure?": Preliminary Insights from Scaling Product Comparisons to Multiple Shops
• Cluster Analysis with Deep Embeddings and Contrastive Learning
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck
• Revisiting Deep Generalized Canonical Correlation Analysis
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• Self-Supervised Learning for Fine-Grained Visual Categorization
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
• An Empirical Study of Graph Contrastive Learning
• TailorMe: Self-Supervised Learning of an Anatomically Constrained Volumetric Human Shape Model
• Compressive Visual Representations
• A Histopathology Study Comparing Contrastive Semi-Supervised and Fully Supervised Learning
• MACK: Mismodeling Addressed with Contrastive Knowledge
• Semi-weakly Supervised Contrastive Representation Learning for Retinal Fundus Images
• Information Theoretic Representation Distillation
• Contrastive Learning of Global-Local Video Representations
• Unsupervised learning on spontaneous retinal activity leads to efficient neural representation geometry
• Transfer Learning with Pre-trained Conditional Generative Models
• Nonequilibrium thermodynamics of self-supervised learning
• Learning Representations for Pixel-based Control: What Matters and Why?
• On the robustness of self-supervised representations for multi-view object classification
• Graph Self-Supervised Learning with Learnable Structural and Positional Encodings
• Unsupervised End-to-End Training with a Self-Defined Target
• Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
• Learning Representations on the Unit Sphere: Investigating Angular Gaussian and von Mises-Fisher Distributions for Online Continual Learning
• SERE: Exploring Feature Self-relation for Self-supervised Transformer
• Data-Driven Self-Supervised Graph Representation Learning
• PatchGame: Learning to Signal Mid-level Patches in Referential Games
• Stochastic Contrastive Learning
• HoughCL: Finding Better Positive Pairs in Dense Self-supervised Learning
• MIO : Mutual Information Optimization using Self-Supervised Binary Contrastive Learning
• Equivariant Contrastive Learning
• Vision Pair Learning: An Efficient Training Framework for Image Classification
• GenURL: A General Framework for Unsupervised Representation Learning
• Barlow Graph Auto-Encoder for Unsupervised Network Embedding
• Domain-Agnostic Clustering with Self-Distillation
• Learning Deep Representation with Energy-Based Self-Expressiveness for Subspace Clustering
• Deep Bregman Divergence for Contrastive Learning of Visual Representations
• Leveraging Domain Adaptation for Low-Resource Geospatial Machine Learning
• RRLFSOR: An Efficient Self-Supervised Learning Strategy of Graph Convolutional Networks
• On-target Adaptation
• Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection
• Self-Supervised Neural Architecture Search for Imbalanced Datasets
• Sparsity-Probe: Analysis tool for Deep Learning Models
• Evaluating the fairness of fine-tuning strategies in self-supervised learning
• Natural Attribute-based Shift Detection
• Self-Supervised Visual Representation Learning Using Lightweight Architectures
• Self-Verification in Image Denoising