Exploring Simple Siamese Representation Learning

Abstract

Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code will be made available.

arXiv commentTechnical report, 10 pages

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Unsupervised Embedding Learning via Invariant and Spreading Instance Feature

Cited by in corpus (120)

• BEiT: BERT Pre-Training of Image Transformers
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning
• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
• Hard Negative Mixing for Contrastive Learning
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• Graph Barlow Twins: A self-supervised representation learning framework for graphs
• Cluster-guided Asymmetric Contrastive Learning for Unsupervised Person Re-Identification
• Self-Supervised Multisensor Change Detection
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• 3D Infomax improves GNNs for Molecular Property Prediction
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• Aligning Pretraining for Detection via Object-Level Contrastive Learning
• Dataset Condensation with Differentiable Siamese Augmentation
• Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks
• Whitening for Self-Supervised Representation Learning
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Contrastive Learning with Stronger Augmentations
• Time-Series Representation Learning via Temporal and Contextual Contrasting
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Understanding Self-supervised Learning with Dual Deep Networks
• Self-supervised Contrastive Learning of Multi-view Facial Expressions
• Contrastive Model Inversion for Data-Free Knowledge Distillation
• Cross-Camera Feature Prediction for Intra-Camera Supervised Person Re-identification across Distant Scenes
• Self-Supervised Learning with Kernel Dependence Maximization
• Towards a Unified Framework for Fair and Stable Graph Representation Learning
• Self-supervised EEG Representation Learning for Automatic Sleep Staging
• Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition
• Improving Weakly Supervised Temporal Action Localization by Exploiting Multi-resolution Information in Temporal Domain
• Pre-Trained Models: Past, Present and Future
• Learning From Long-Tailed Data With Noisy Labels
• A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Multiple Domain Experts Collaborative Learning: Multi-Source Domain Generalization For Person Re-Identification
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Self-supervised Remote Sensing Images Change Detection at Pixel-level
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
• Rethinking Self-Supervised Learning: Small is Beautiful
• Self-Guided Contrastive Learning for BERT Sentence Representations
• Self-supervised Graph Neural Networks without explicit negative sampling
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Improving Transferability of Representations via Augmentation-Aware Self-Supervision
• Interpretable agent communication from scratch (with a generic visual processor emerging on the side)
• DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning
• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Support-Set Based Cross-Supervision for Video Grounding
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks
• Unsupervised Object-Level Representation Learning from Scene Images
• Focus Your Distribution: Coarse-to-Fine Non-Contrastive Learning for Anomaly Detection and Localization
• Winning the CVPR'2021 Kinetics-GEBD Challenge: Contrastive Learning Approach
• Set Representation Learning with Generalized Sliced-Wasserstein Embeddings
• Domain Adapting Ability of Self-Supervised Learning for Face Recognition
• SimTriplet: Simple Triplet Representation Learning with a Single GPU
• Unsupervised domain adaptation via coarse-to-fine feature alignment method using contrastive learning
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• On Feature Decorrelation in Self-Supervised Learning
• DyFormer: A Scalable Dynamic Graph Transformer with Provable Benefits on Generalization Ability
• Dense Semantic Contrast for Self-Supervised Visual Representation Learning
• Are Negative Samples Necessary in Entity Alignment? An Approach with High Performance, Scalability and Robustness
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
• Imbalance-Aware Self-Supervised Learning for 3D Radiomic Representations
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• Motion-aware Contrastive Video Representation Learning via Foreground-background Merging
• Polyphone Disambiguation in Mandarin Chinese with Semi-Supervised Learning
• Scale-Equivariant Imaging: Self-Supervised Learning for Image Super-Resolution and Deblurring
• An Empirical Study of Graph Contrastive Learning
• TENET: A Time-reversal Enhancement Network for Noise-robust ASR
• CMV-BERT: Contrastive multi-vocab pretraining of BERT
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• Structure-Aware Hard Negative Mining for Heterogeneous Graph Contrastive Learning
• Unsupervised Document Embedding via Contrastive Augmentation
• Visualizing MuZero Models
• Contrastive Learning of Global-Local Video Representations
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Addressing Feature Suppression in Unsupervised Visual Representations
• Tasting the cake: evaluating self-supervised generalization on out-of-distribution multimodal MRI data
• Self-supervised Change Detection in Multi-view Remote Sensing Images
• Long-Short Temporal Contrastive Learning of Video Transformers
• Exploring Set Similarity for Dense Self-supervised Representation Learning
• UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning
• AlphaMatch: Improving Consistency for Semi-supervised Learning with Alpha-divergence
• Temporal Knowledge Consistency for Unsupervised Visual Representation Learning
• Jointly Learnable Data Augmentations for Self-Supervised GNNs
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene
• Contrastive Adaptive Propagation Graph Neural Networks for Efficient Graph Learning
• Object-Aware Cropping for Self-Supervised Learning
• CoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised Learning
• ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
• Unsupervised Visual Representation Learning by Online Constrained K-Means
• Characterizing and Improving the Robustness of Self-Supervised Learning through Background Augmentations
• GraphVICRegHSIC: Towards improved self-supervised representation learning for graphs with a hyrbid loss function
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay
• Provable Lifelong Learning of Representations
• Contrastive Feature Loss for Image Prediction
• Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation
• Examination of PCA Utilisation for Multilabel Classifier of Multispectral Images
• Data-Driven Self-Supervised Graph Representation Learning
• Vision Pair Learning: An Efficient Training Framework for Image Classification
• Voxel-level Siamese Representation Learning for Abdominal Multi-Organ Segmentation
• On-target Adaptation
• Voxel-wise Cross-Volume Representation Learning for 3D Neuron Reconstruction
• Self-Supervised Neural Architecture Search for Imbalanced Datasets
• Homography augumented momentum constrastive learning for SAR image retrieval
• Long Short View Feature Decomposition via Contrastive Video Representation Learning
• Towards Universal Dense Retrieval for Open-domain Question Answering
• Learning Rich Nearest Neighbor Representations from Self-supervised Ensembles
• PatchGame: Learning to Signal Mid-level Patches in Referential Games
• SVMax: A Feature Embedding Regularizer