i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning

Abstract

Contrastive representation learning has shown to be effective to learn representations from unlabeled data. However, much progress has been made in vision domains relying on data augmentations carefully designed using domain knowledge. In this work, we propose i-Mix, a simple yet effective domain-agnostic regularization strategy for improving contrastive representation learning. We cast contrastive learning as training a non-parametric classifier by assigning a unique virtual class to each data in a batch. Then, data instances are mixed in both the input and virtual label spaces, providing more augmented data during training. In experiments, we demonstrate that i-Mix consistently improves the quality of learned representations across domains, including image, speech, and tabular data. Furthermore, we confirm its regularization effect via extensive ablation studies across model and dataset sizes. The code is available at https://github.com/kibok90/imix.

arXiv commentICLR 2021

References in corpus (29)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• A Simple Framework for Contrastive Learning of Visual Representations
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• AutoAugment: Learning Augmentation Policies from Data
• What Makes for Good Views for Contrastive Learning?
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Augmenting Data with Mixup for Sentence Classification: An Empirical Study
• Supervised Contrastive Learning
• FMix: Enhancing Mixed Sample Data Augmentation
• Incremental Classifier Learning with Generative Adversarial Networks
• Unsupervised Embedding Learning via Invariant and Spreading Instance Feature
• No Fuss Distance Metric Learning using Proxies
• MixCo: Mix-up Contrastive Learning for Visual Representation
• Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks
• Mixed batches and symmetric discriminators for GAN training
• On Adversarial Mixup Resynthesis
• Towards Domain-Agnostic Contrastive Learning

Cited by in corpus (9)

• Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification
• Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup
• Contrastive-mixup learning for improved speaker verification
• It Takes Two to Tango: Mixup for Deep Metric Learning
• Supporting Clustering with Contrastive Learning
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling
• Hybrid Generative-Contrastive Representation Learning
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• Equivariant Contrastive Learning