Self-Supervised Learning with Kernel Dependence Maximization

Abstract

We approach self-supervised learning of image representations from a statistical dependence perspective, proposing Self-Supervised Learning with the Hilbert-Schmidt Independence Criterion (SSL-HSIC). SSL-HSIC maximizes dependence between representations of transformations of an image and the image identity, while minimizing the kernelized variance of those representations. This framework yields a new understanding of InfoNCE, a variational lower bound on the mutual information (MI) between different transformations. While the MI itself is known to have pathologies which can result in learning meaningless representations, its bound is much better behaved: we show that it implicitly approximates SSL-HSIC (with a slightly different regularizer). Our approach also gives us insight into BYOL, a negative-free SSL method, since SSL-HSIC similarly learns local neighborhoods of samples. SSL-HSIC allows us to directly optimize statistical dependence in time linear in the batch size, without restrictive data assumptions or indirect mutual information estimators. Trained with or without a target network, SSL-HSIC matches the current state-of-the-art for standard linear evaluation on ImageNet, semi-supervised learning and transfer to other classification and vision tasks such as semantic segmentation, depth estimation and object recognition. Code is available at https://github.com/deepmind/ssl_hsic .

References in corpus (10)

• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Exploring Simple Siamese Representation Learning
• Training Neural Networks with Local Error Signals
• Intriguing Properties of Contrastive Losses
• Contrastive Learning Inverts the Data Generating Process
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Information Condensing Active Learning

Cited by in corpus (5)

• Bilateral Dependency Optimization: Defending Against Model-inversion Attacks
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck
• Deep Bregman Divergence for Contrastive Learning of Visual Representations
• Pretext Tasks selection for multitask self-supervised speech representation learning