A critical analysis of self-supervision, or what we can learn from a single image

Abstract

We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels. We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training. We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.

Accepted paper at the International Conference on Learning Representations (ICLR) 2020

References in corpus (3)

• How transferable are features in deep neural networks?
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• What makes ImageNet good for transfer learning?

Cited by in corpus (23)

• A Simple Framework for Contrastive Learning of Visual Representations
• What Makes for Good Views for Contrastive Learning?
• Hard Negative Mixing for Contrastive Learning
• Self-labelling via simultaneous clustering and representation learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning
• Deep Learning for Insider Threat Detection: Review, Challenges and Opportunities
• When Does Self-supervision Improve Few-shot Learning?
• Negative Data Augmentation
• One-Shot Unsupervised Cross-Domain Detection
• Towards Defending Multiple ${\ell }_p$-norm Bounded Adversarial Perturbations via Gated Batch Normalization
• Contrastive Neural Processes for Self-Supervised Learning
• Fine-grained Anomaly Detection via Multi-task Self-Supervision
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
• Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
• Don't miss the Mismatch: Investigating the Objective Function Mismatch for Unsupervised Representation Learning
• A Unified Mixture-View Framework for Unsupervised Representation Learning
• Efficient Estimation of Influence of a Training Instance
• Learning to See by Looking at Noise
• Segmentation of VHR EO Images using Unsupervised Learning
• Weakly Supervised Recovery of Semantic Attributes
• Hybrid BYOL-ViT: Efficient approach to deal with small datasets
• On Equivariant and Invariant Learning of Object Landmark Representations