On the robustness of self-supervised representations for multi-view object classification

Abstract

It is known that representations from self-supervised pre-training can perform on par, and often better, on various downstream tasks than representations from fully-supervised pre-training. This has been shown in a host of settings such as generic object classification and detection, semantic segmentation, and image retrieval. However, some issues have recently come to the fore that demonstrate some of the failure modes of self-supervised representations, such as performance on non-ImageNet-like data, or complex scenes. In this paper, we show that self-supervised representations based on the instance discrimination objective lead to better representations of objects that are more robust to changes in the viewpoint and perspective of the object. We perform experiments of modern self-supervised methods against multiple supervised baselines to demonstrate this, including approximating object viewpoint variation through homographies, and real-world tests based on several multi-view datasets. We find that self-supervised representations are more robust to object viewpoint and appear to encode more pertinent information about objects that facilitate the recognition of objects from novel views.

arXiv commentPublished in Pattern Recognition Letters

References in corpus (13)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Representation Learning by Predicting Image Rotations
• Fine-Grained Visual Classification of Aircraft
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Deep Clustering for Unsupervised Learning of Visual Features
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• Parametric Instance Classification for Unsupervised Visual Feature Learning