Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination

Abstract

Unsupervised feature learning has made great strides with contrastive learning based on instance discrimination and invariant mapping, as benchmarked on curated class-balanced datasets. However, natural data could be highly correlated and long-tail distributed. Natural between-instance similarity conflicts with the presumed instance distinction, causing unstable training and poor performance. Our idea is to discover and integrate between-instance similarity into contrastive learning, not directly by instance grouping, but by cross-level discrimination (CLD) between instances and local instance groups. While invariant mapping of each instance is imposed by attraction within its augmented views, between-instance similarity could emerge from common repulsion against instance groups. Our batch-wise and cross-view comparisons also greatly improve the positive/negative sample ratio of contrastive learning and achieve better invariant mapping. To effect both grouping and discrimination objectives, we impose them on features separately derived from a shared representation. In addition, we propose normalized projection heads and unsupervised hyper-parameter tuning for the first time. Our extensive experimentation demonstrates that CLD is a lean and powerful add-on to existing methods such as NPID, MoCo, InfoMin, and BYOL on highly correlated, long-tail, or balanced datasets. It not only achieves new state-of-the-art on self-supervision, semi-supervision, and transfer learning benchmarks, but also beats MoCo v2 and SimCLR on every reported performance attained with a much larger compute. CLD effectively brings unsupervised learning closer to natural data and real-world applications. Our code is publicly available at: https://github.com/frank-xwang/CLD-UnsupervisedLearning.

arXiv commentAccepted at CVPR 2021; Project page: http://people.eecs.berkeley.edu/~xdwang/projects/CLD/

References in corpus (20)

• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Prototypical Contrastive Learning of Unsupervised Representations
• Exploring Simple Siamese Representation Learning
• Large Scale Adversarial Representation Learning
• On Mutual Information Maximization for Representation Learning
• Joint Unsupervised Learning of Deep Representations and Image Clusters
• Unsupervised Embedding Learning via Invariant and Spreading Instance Feature
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Large-Scale Long-Tailed Recognition in an Open World
• Colorization as a Proxy Task for Visual Understanding
• Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
• Self-Supervised Feature Learning by Learning to Spot Artifacts
• Self-Supervised Learning via Conditional Motion Propagation

Cited by in corpus (2)

• Contrastive Learning with Stronger Augmentations
• Unsupervised Visual Attention and Invariance for Reinforcement Learning