A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
arXiv:2104.13712
Abstract
In this report, we relate the algorithmic design of Barlow Twins' method to the Hilbert-Schmidt Independence Criterion (HSIC), thus establishing it as a contrastive learning approach that is free of negative samples. Through this perspective, we argue that Barlow Twins (and thus the class of negative-sample-free contrastive learning methods) suggests a possibility to bridge the two major families of self-supervised learning philosophies: non-contrastive and contrastive approaches. In particular, Barlow twins exemplified how we could combine the best practices of both worlds: avoiding the need of large training batch size and negative sample pairing (like non-contrastive methods) and avoiding symmetry-breaking network designs (like contrastive methods).
References in corpus (6)
- Bootstrap your own latent: A new approach to self-supervised Learning
- The information bottleneck method
- Learning deep representations by mutual information estimation and maximization
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- Exploring Simple Siamese Representation Learning
- Wasserstein Dependency Measure for Representation Learning
Cited by in corpus (5)
- Graph Barlow Twins: A self-supervised representation learning framework for graphs
- COCOA: Cross Modality Contrastive Learning for Sensor Data
- DECAR: Deep Clustering for learning general-purpose Audio Representations
- An Empirical Study of Graph Contrastive Learning
- GraphVICRegHSIC: Towards improved self-supervised representation learning for graphs with a hyrbid loss function