Towards Domain-Agnostic Contrastive Learning

Abstract

Despite recent success, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation. To overcome such limitation, we propose a novel domain-agnostic approach to contrastive learning, named DACL, that is applicable to domains where invariances, and thus, data augmentation techniques, are not readily available. Key to our approach is the use of Mixup noise to create similar and dissimilar examples by mixing data samples differently either at the input or hidden-state levels. To demonstrate the effectiveness of DACL, we conduct experiments across various domains such as tabular data, images, and graphs. Our results show that DACL not only outperforms other domain-agnostic noising methods, such as Gaussian-noise, but also combines well with domain-specific methods, such as SimCLR, to improve self-supervised visual representation learning. Finally, we theoretically analyze our method and show advantages over the Gaussian-noise based contrastive learning approach.

arXiv commentPublished in ICML 2021

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Interpolation Consistency Training for Semi-Supervised Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Semi-supervised Sequence Learning
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Large Scale Adversarial Representation Learning
• How Powerful are Graph Neural Networks?
• Strategies for Pre-training Graph Neural Networks
• Augmenting Data with Mixup for Sentence Classification: An Empirical Study
• Selfie: Self-supervised Pretraining for Image Embedding
• Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer
• MixCo: Mix-up Contrastive Learning for Visual Representation
• Interpolation-based semi-supervised learning for object detection
• GraphMix: Improved Training of GNNs for Semi-Supervised Learning

Cited by in corpus (4)

• i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• An Empirical Study of Graph Contrastive Learning
• Equivariant Contrastive Learning