DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learning

Abstract

Self-supervised learning (SSL) has gained remarkable success, for which contrastive learning (CL) plays a key role. However, the recent development of new non-CL frameworks has achieved comparable or better performance with high improvement potential, prompting researchers to enhance these frameworks further. Assimilating CL into non-CL frameworks has been thought to be beneficial, but empirical evidence indicates no visible improvements. In view of that, this paper proposes a strategy of performing CL along the dimensional direction instead of along the batch direction as done in conventional contrastive learning, named Dimensional Contrastive Learning (DimCL). DimCL aims to enhance the feature diversity, and it can serve as a regularizer to prior SSL frameworks. DimCL has been found to be effective, and the hardness-aware property is identified as a critical reason for its success. Extensive experimental results reveal that assimilating DimCL into SSL frameworks leads to performance improvement by a non-trivial margin on various datasets and backbone architectures.

References in corpus (18)

• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Representation Learning with Contrastive Predictive Coding
• Bootstrap your own latent: A new approach to self-supervised Learning
• In Defense of the Triplet Loss for Person Re-Identification
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Learning Representations by Maximizing Mutual Information Across Views
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
• XCiT: Cross-Covariance Image Transformers
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• 3D Face Reconstruction from A Single Image Assisted by 2D Face Images in the Wild
• Efficient Self-supervised Vision Transformers for Representation Learning
• Contrastive Learning with Adversarial Examples
• Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning