Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
arXiv:2102.06866
Abstract
Instance discriminative self-supervised representation learning has been attracted attention thanks to its unsupervised nature and informative feature representation for downstream tasks. In practice, it commonly uses a larger number of negative samples than the number of supervised classes. However, there is an inconsistency in the existing analysis; theoretically, a large number of negative samples degrade classification performance on a downstream supervised task, while empirically, they improve the performance. We provide a novel framework to analyze this empirical result regarding negative samples using the coupon collector's problem. Our bound can implicitly incorporate the supervised loss of the downstream task in the self-supervised loss by increasing the number of negative samples. We confirm that our proposed analysis holds on real-world benchmark datasets.
NeurIPS 2021. 26 pages, 6 figures, and 6 tables
References in corpus (12)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Contrastive Representation Learning: A Framework and Review
- Skip-Thought Vectors
- Large Batch Training of Convolutional Networks
- A Theoretical Analysis of Contrastive Unsupervised Representation Learning
- Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
- Predicting What You Already Know Helps: Provable Self-Supervised Learning
- One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities
- SimCSE: Simple Contrastive Learning of Sentence Embeddings
- Self-Supervised Learning with Kernel Dependence Maximization
- Investigating the Role of Negatives in Contrastive Representation Learning