For self-supervised learning, Rationality implies generalization, provably

Abstract

We prove a new upper bound on the generalization gap of classifiers that are obtained by first using self-supervision to learn a representation $r$ of the training data, and then fitting a simple (e.g., linear) classifier $g$ to the labels. Specifically, we show that (under the assumptions described below) the generalization gap of such classifiers tends to zero if $\mathsf{C}(g)\ll n$, where $\mathsf{C}(g)$ is an appropriately-defined measure of the simple classifier $g$'s complexity, and $n$ is the number of training samples. We stress that our bound is independent of the complexity of the representation $r$. We do not make any structural or conditional-independence assumptions on the representation-learning task, which can use the same training dataset that is later used for classification. Rather, we assume that the training procedure satisfies certain natural noise-robustness (adding small amount of label noise causes small degradation in performance) and rationality (getting the wrong label is not better than getting no label at all) conditions that widely hold across many standard architectures. We show that our bound is non-vacuous for many popular representation-learning based classifiers on CIFAR-10 and ImageNet, including SimCLR, AMDIM and MoCo.

References in corpus (22)

• A Simple Framework for Contrastive Learning of Visual Representations
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Language Models are Few-Shot Learners
• Improved Baselines with Momentum Contrastive Learning
• Wide Residual Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Large Scale Adversarial Representation Learning
• Noise Tolerance under Risk Minimization
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Uniform convergence may be unable to explain generalization in deep learning
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Reasoning About Generalization via Conditional Mutual Information
• Stability Analysis for Regularized Least Squares Regression

Cited by in corpus (2)

• Data Compression and Inference in Cosmology with Self-Supervised Machine Learning
• Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis