PAC-Bayesian Contrastive Unsupervised Representation Learning

Abstract

Contrastive unsupervised representation learning (CURL) is the state-of-the-art technique to learn representations (as a set of features) from unlabelled data. While CURL has collected several empirical successes recently, theoretical understanding of its performance was still missing. In a recent work, Arora et al. (2019) provide the first generalisation bounds for CURL, relying on a Rademacher complexity. We extend their framework to the flexible PAC-Bayes setting, allowing us to deal with the non-iid setting. We present PAC-Bayesian generalisation bounds for CURL, which are then used to derive a new representation learning algorithm. Numerical experiments on real-life datasets illustrate that our algorithm achieves competitive accuracy, and yields non-vacuous generalisation bounds.

arXiv commentPublished in the proceedings of the Conference on Uncertainty in Artificial Intelligence 2020 (UAI)

References in corpus (9)

• Distributed Representations of Words and Phrases and their Compositionality
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• A Simple Framework for Contrastive Learning of Visual Representations
• An efficient framework for learning sentence representations
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• A Primer on PAC-Bayesian Learning
• Simpler PAC-Bayesian Bounds for Hostile Data

Cited by in corpus (4)

• User-friendly introduction to PAC-Bayes bounds
• Learning via Wasserstein-Based High Probability Generalisation Bounds
• Functional Regularization for Representation Learning: A Unified Theoretical Perspective
• A note on regularised NTK dynamics with an application to PAC-Bayesian training