Self-Supervised Relational Reasoning for Representation Learning

Abstract

In self-supervised learning, a system is tasked with achieving a surrogate objective by defining alternative targets on a set of unlabeled data. The aim is to build useful representations that can be used in downstream tasks, without costly manual annotation. In this work, we propose a novel self-supervised formulation of relational reasoning that allows a learner to bootstrap a signal from information implicit in unlabeled data. Training a relation head to discriminate how entities relate to themselves (intra-reasoning) and other entities (inter-reasoning), results in rich and descriptive representations in the underlying neural network backbone, which can be used in downstream tasks such as classification and image retrieval. We evaluate the proposed method following a rigorous experimental procedure, using standard datasets, protocols, and backbones. Self-supervised relational reasoning outperforms the best competitor in all conditions by an average 14% in accuracy, and the most recent state-of-the-art model by 3%. We link the effectiveness of the method to the maximization of a Bernoulli log-likelihood, which can be considered as a proxy for maximizing the mutual information, resulting in a more efficient objective with respect to the commonly used contrastive losses.

arXiv commentAdvances in Neural Information Processing Systems (NeurIPS 2020, Spotlight)

References in corpus (12)

• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Improved Regularization of Convolutional Neural Networks with Cutout
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Learning deep representations by mutual information estimation and maximization
• Momentum Contrast for Unsupervised Visual Representation Learning
• Contrastive Multiview Coding
• On Mutual Information Maximization for Representation Learning
• Supervised Contrastive Learning
• Discovering objects and their relations from entangled scene representations
• Learning Independent Features with Adversarial Nets for Non-linear ICA
• Defining Benchmarks for Continual Few-Shot Learning

Cited by in corpus (6)

• Self-learning for weakly supervised Gleason grading of local patterns
• Self-Supervised Time Series Representation Learning by Inter-Intra Relational Reasoning
• Learning Online Visual Invariances for Novel Objects via Supervised and Self-Supervised Training
• Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting
• Don't miss the Mismatch: Investigating the Objective Function Mismatch for Unsupervised Representation Learning
• MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning