Barlow Twins: Self-Supervised Learning via Redundancy Reduction

Abstract

Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.

13 pages, 6 figures, to appear at ICML 2021

References in corpus (3)

• Large Batch Training of Convolutional Networks
• Exploring Simple Siamese Representation Learning
• Understanding self-supervised Learning Dynamics without Contrastive Pairs

Cited by in corpus (120)

• Graph Self-Supervised Learning: A Survey
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Towards Representation Alignment and Uniformity in Collaborative Filtering
• Video Transformers: A Survey
• Twin Contrastive Learning for Online Clustering
• Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structure
• Graph Barlow Twins: A self-supervised representation learning framework for graphs
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding
• CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification
• Learning Disentangled Representations in the Imaging Domain
• SelfCF: A Simple Framework for Self-supervised Collaborative Filtering
• Understanding Dimensional Collapse in Contrastive Self-supervised Learning
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• 3D Infomax improves GNNs for Molecular Property Prediction
• Large-scale Unsupervised Semantic Segmentation
• Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast
• CLIP in Medical Imaging: A Survey
• DualNet: Continual Learning, Fast and Slow
• Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis
• Aligning Pretraining for Detection via Object-Level Contrastive Learning
• Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Inter-Domain Mixup for Semi-Supervised Domain Adaptation
• Do Different Tracking Tasks Require Different Appearance Models?
• A Study of the Generalizability of Self-Supervised Representations
• Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence
• False: False Negative Samples Aware Contrastive Learning for Semantic Segmentation of High-Resolution Remote Sensing Image
• Self-supervised visual learning in the low-data regime: a comparative evaluation
• A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis
• Self-Supervised Training Enhances Online Continual Learning
• Self-Supervised Learning with Kernel Dependence Maximization
• Towards the Generalization of Contrastive Self-Supervised Learning
• GaitFormer: Learning Gait Representations with Noisy Multi-Task Learning
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• Contrastive Learning with Positive-Negative Frame Mask for Music Representation
• Foundation Models and Transformers for Anomaly Detection: A Survey
• Large-Scale Hyperspectral Image Clustering Using Contrastive Learning
• Universum-inspired Supervised Contrastive Learning
• Learning From Long-Tailed Data With Noisy Labels
• Self-supervised learning unveils change in urban housing from street-level images
• Contrastive Learning of Person-independent Representations for Facial Action Unit Detection
• A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Test time Adaptation through Perturbation Robustness
• Semi-supervised Open-World Object Detection
• Temperature as Uncertainty in Contrastive Learning
• Non-contrastive representation learning for intervals from well logs
• Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learning
• Rethinking Supervised Pre-training for Better Downstream Transferring
• DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning
• A Softmax-free Loss Function Based on Predefined Optimal-distribution of Latent Features for Deep Learning Classifier
• SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives
• Self-Supervised Learning for Text Recognition: A Critical Survey
• MUSE: Music Recommender System with Shuffle Play Recommendation Enhancement
• Local Contrastive Feature learning for Tabular Data
• Unsupervised Pre-Training for 3D Leaf Instance Segmentation
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
• INoD: Injected Noise Discriminator for Self-Supervised Representation Learning in Agricultural Fields
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• Self-Supervised Learning by Estimating Twin Class Distributions
• Contrastive Multi-View Textual-Visual Encoding: Towards One Hundred Thousand-Scale One-Shot Logo Identification
• TURBO: The Swiss Knife of Auto-Encoders
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• CUCL: Codebook for Unsupervised Continual Learning
• Contrastive Self-Supervised Learning for Spatio-Temporal Analysis of Lung Ultrasound Videos
• Self-Supervised Representation Learning for Nerve Fiber Distribution Patterns in 3D-PLI
• Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
• Self-Supervised Pre-Training Boosts Semantic Scene Segmentation on LiDAR Data
• MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• Revisiting Deep Generalized Canonical Correlation Analysis
• Cluster Analysis with Deep Embeddings and Contrastive Learning
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• Self-Supervised Learning for Fine-Grained Visual Categorization
• Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck
• An Empirical Study of Graph Contrastive Learning
• "Are you sure?": Preliminary Insights from Scaling Product Comparisons to Multiple Shops
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
• Transfer Learning with Pre-trained Conditional Generative Models
• MACK: Mismodeling Addressed with Contrastive Knowledge
• Unsupervised learning on spontaneous retinal activity leads to efficient neural representation geometry
• TailorMe: Self-Supervised Learning of an Anatomically Constrained Volumetric Human Shape Model
• Information Theoretic Representation Distillation
• A Histopathology Study Comparing Contrastive Semi-Supervised and Fully Supervised Learning
• Compressive Visual Representations
• Semi-weakly Supervised Contrastive Representation Learning for Retinal Fundus Images
• Contrastive Learning of Global-Local Video Representations
• Unsupervised End-to-End Training with a Self-Defined Target
• Nonequilibrium thermodynamics of self-supervised learning
• Learning Representations for Pixel-based Control: What Matters and Why?
• On the robustness of self-supervised representations for multi-view object classification
• Graph Self-Supervised Learning with Learnable Structural and Positional Encodings
• Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
• Learning Deep Representation with Energy-Based Self-Expressiveness for Subspace Clustering
• Leveraging Domain Adaptation for Low-Resource Geospatial Machine Learning
• Self-Verification in Image Denoising
• PatchGame: Learning to Signal Mid-level Patches in Referential Games
• Vision Pair Learning: An Efficient Training Framework for Image Classification
• Deep Bregman Divergence for Contrastive Learning of Visual Representations
• Self-Supervised Visual Representation Learning Using Lightweight Architectures
• RRLFSOR: An Efficient Self-Supervised Learning Strategy of Graph Convolutional Networks
• On-target Adaptation
• Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection
• Self-Supervised Neural Architecture Search for Imbalanced Datasets
• Learning Representations on the Unit Sphere: Investigating Angular Gaussian and von Mises-Fisher Distributions for Online Continual Learning
• Evaluating the fairness of fine-tuning strategies in self-supervised learning
• Sparsity-Probe: Analysis tool for Deep Learning Models
• Data-Driven Self-Supervised Graph Representation Learning
• Natural Attribute-based Shift Detection
• MIO : Mutual Information Optimization using Self-Supervised Binary Contrastive Learning
• Stochastic Contrastive Learning
• HoughCL: Finding Better Positive Pairs in Dense Self-supervised Learning
• Equivariant Contrastive Learning
• GenURL: A General Framework for Unsupervised Representation Learning
• Barlow Graph Auto-Encoder for Unsupervised Network Embedding
• Domain-Agnostic Clustering with Self-Distillation
• SERE: Exploring Feature Self-relation for Self-supervised Transformer