Improving Contrastive Learning by Visualizing Feature Transformation

Abstract

Contrastive learning, which aims at minimizing the distance between positive pairs while maximizing that of negative ones, has been widely and successfully applied in unsupervised feature learning, where the design of positive and negative (pos/neg) pairs is one of its keys. In this paper, we attempt to devise a feature-level data manipulation, differing from data augmentation, to enhance the generic contrastive self-supervised learning. To this end, we first design a visualization scheme for pos/neg score (Pos/neg score indicates cosine similarity of pos/neg pair.) distribution, which enables us to analyze, interpret and understand the learning process. To our knowledge, this is the first attempt of its kind. More importantly, leveraging this tool, we gain some significant observations, which inspire our novel Feature Transformation proposals including the extrapolation of positives. This operation creates harder positives to boost the learning because hard positives enable the model to be more view-invariant. Besides, we propose the interpolation among negatives, which provides diversified negatives and makes the model more discriminative. It is the first attempt to deal with both challenges simultaneously. Experiment results show that our proposed Feature Transformation can improve at least 6.0% accuracy on ImageNet-100 over MoCo baseline, and about 2.0% accuracy on ImageNet-1K over the MoCoV2 baseline. Transferring to the downstream tasks successfully demonstrate our model is less task-bias. Visualization tools and codes https://github.com/DTennant/CL-Visualizing-Feature-Transformation .

arXiv commentICCV 2021(Oral), supplementary materials included. Codes and visualization tools: https://github.com/DTennant/CL-Visualizing-Feature-Transformation

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Fine-Grained Visual Classification of Aircraft
• What Makes for Good Views for Contrastive Learning?
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Supervised Contrastive Learning
• MixCo: Mix-up Contrastive Learning for Visual Representation