Automatic Shortcut Removal for Self-Supervised Representation Learning

Abstract

In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply, without human annotation. A central challenge in this approach is that the feature extractor quickly learns to exploit low-level visual features such as color aberrations or watermarks and then fails to learn useful semantic representations. Much work has gone into identifying such "shortcut" features and hand-designing schemes to reduce their effect. Here, we propose a general framework for mitigating the effect shortcut features. Our key assumption is that those features which are the first to be exploited for solving the pretext task may also be the most vulnerable to an adversary trained to make the task harder. We show that this assumption holds across common pretext tasks and datasets by training a "lens" network to make small image changes that maximally reduce performance in the pretext task. Representations learned with the modified images outperform those learned without in all tested cases. Additionally, the modifications made by the lens reveal how the choice of pretext task and dataset affects the features learned by self-supervision.

References in corpus (10)

• Representation Learning with Contrastive Predictive Coding
• Learning deep representations by mutual information estimation and maximization
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Contrastive Multiview Coding
• On Mutual Information Maximization for Representation Learning
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Improvements to context based self-supervised learning
• Image Synthesis with a Single (Robust) Classifier

Cited by in corpus (8)

• Shortcut Learning in Deep Neural Networks
• What Makes for Good Views for Contrastive Learning?
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• Benign Shortcut for Debiasing: Fair Visual Recognition via Intervention with Shortcut Features
• Geographical Knowledge-driven Representation Learning for Remote Sensing Images
• Generator Versus Segmentor: Pseudo-healthy Synthesis
• Achieving Domain Robustness in Stereo Matching Networks by Removing Shortcut Learning