SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
arXiv:2106.15147
Abstract
Self-supervised contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-art performance with orders of magnitude less labeled data. However, such methods are domain-specific and little has been done to leverage this technique on real-world tabular datasets. We propose SCARF, a simple, widely-applicable technique for contrastive learning, where views are formed by corrupting a random subset of features. When applied to pre-train deep neural networks on the 69 real-world, tabular classification datasets from the OpenML-CC18 benchmark, SCARF not only improves classification accuracy in the fully-supervised setting but does so also in the presence of label noise and in the semi-supervised setting where only a fraction of the available training data is labeled. We show that SCARF complements existing strategies and outperforms alternatives like autoencoders. We conduct comprehensive ablations, detailing the importance of a range of factors.
arXiv commentICLR 2022 Spotlight
References in corpus (35)
- Distributed Representations of Words and Phrases and their Compositionality
- Distilling the Knowledge in a Neural Network
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Natural Language Processing (almost) from Scratch
- mixup: Beyond Empirical Risk Minimization
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Bootstrap your own latent: A new approach to self-supervised Learning
- Language Models are Few-Shot Learners
- The Effectiveness of Data Augmentation in Image Classification using Deep Learning
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Learning deep representations by mutual information estimation and maximization
- Cross-lingual Language Model Pretraining
- Pre-trained Models for Natural Language Processing: A Survey
- OpenML: networked science in machine learning
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- When Does Label Smoothing Help?
- Adversarial Feature Learning
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Contrastive Multiview Coding
- MPNet: Masked and Permuted Pre-training for Language Understanding
- Contrastive Multi-View Representation Learning on Graphs
- UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
- BERTje: A Dutch BERT Model
- Unsupervised Learning by Predicting Noise
- Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
- A Bayesian Data Augmentation Approach for Learning Deep Models
- Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
- Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
- Self-supervised Learning for Large-scale Item Recommendations
- Robust Bi-Tempered Logistic Loss Based on Bregman Divergences
- Viewmaker Networks: Learning Views for Unsupervised Representation Learning
- OpenML-Python: an extensible Python API for OpenML
- Self-Supervised GAN to Counter Forgetting