What makes instance discrimination good for transfer learning?

Abstract

Contrastive visual pretraining based on the instance discrimination pretext task has made significant progress. Notably, recent work on unsupervised pretraining has shown to surpass the supervised counterpart for finetuning downstream applications such as object detection and segmentation. It comes as a surprise that image annotations would be better left unused for transfer learning. In this work, we investigate the following problems: What makes instance discrimination pretraining good for transfer learning? What knowledge is actually learned and transferred from these models? From this understanding of instance discrimination, how can we better exploit human annotation labels for pretraining? Our findings are threefold. First, what truly matters for the transfer is low-level and mid-level representations, not high-level representations. Second, the intra-category invariance enforced by the traditional supervised model weakens transferability by increasing task misalignment. Finally, supervised pretraining can be strengthened by following an exemplar-based approach without explicit constraints among the instances within the same category.

arXiv commentAccepted by ICLR 2021

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• How transferable are features in deep neural networks?
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Representation Learning by Predicting Image Rotations
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• What makes ImageNet good for transfer learning?
• Revisiting Self-Supervised Visual Representation Learning
• Distilling Localization for Self-Supervised Representation Learning

Cited by in corpus (12)

• Contrastive Representation Learning: A Framework and Review
• Hard Negative Mixing for Contrastive Learning
• Intriguing Properties of Contrastive Losses
• MixCo: Mix-up Contrastive Learning for Visual Representation
• A Study of the Generalizability of Self-Supervised Representations
• Dual Contrastive Learning for Spatio-temporal Representation
• Rethinking Supervised Pre-training for Better Downstream Transferring
• Instance Localization for Self-supervised Detection Pretraining
• Generative Hierarchical Features from Synthesizing Images
• AutoLTS: Automating Cycling Stress Assessment via Contrastive Learning and Spatial Post-processing
• Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning