Exploring Simple Siamese Representation Learning

Abstract

Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code will be made available.

Technical report, 10 pages

References in corpus (6)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Bootstrap your own latent: A new approach to self-supervised Learning
• Learning Representations by Maximizing Mutual Information Across Views
• Large Batch Training of Convolutional Networks
• Unsupervised Embedding Learning via Invariant and Spreading Instance Feature

Cited by in corpus (14)

• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
• Rethinking Self-Supervised Learning: Small is Beautiful
• XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition
• SimTriplet: Simple Triplet Representation Learning with a Single GPU
• Unsupervised domain adaptation via coarse-to-fine feature alignment method using contrastive learning
• Set Representation Learning with Generalized Sliced-Wasserstein Embeddings
• Unsupervised Document Embedding via Contrastive Augmentation
• Visualizing MuZero Models
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning
• AlphaMatch: Improving Consistency for Semi-supervised Learning with Alpha-divergence
• IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay
• SVMax: A Feature Embedding Regularizer