Functional Regularization for Representation Learning: A Unified Theoretical Perspective

Abstract

Unsupervised and self-supervised learning approaches have become a crucial tool to learn representations for downstream prediction tasks. While these approaches are widely used in practice and achieve impressive empirical gains, their theoretical understanding largely lags behind. Towards bridging this gap, we present a unifying perspective where several such approaches can be viewed as imposing a regularization on the representation via a learnable function using unlabeled data. We propose a discriminative theoretical framework for analyzing the sample complexity of these approaches, which generalizes the framework of (Balcan and Blum, 2010) to allow learnable regularization functions. Our sample complexity bounds show that, with carefully chosen hypothesis classes to exploit the structure in the data, these learnable regularization functions can prune the hypothesis space, and help reduce the amount of labeled data needed. We then provide two concrete examples of functional regularization, one using auto-encoders and the other using masked self-supervision, and apply our framework to quantify the reduction in the sample complexity bound of labeled data. We also provide complementary empirical results to support our analysis.

arXiv commentAccepted at NeurIPS 2020

References in corpus (14)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Understanding deep learning requires rethinking generalization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Supervised Dictionary Learning
• On Mutual Information Maximization for Representation Learning
• Unsupervised Visual Representation Learning by Context Prediction
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Uniform convergence may be unable to explain generalization in deep learning
• Representation Learning: A Review and New Perspectives
• Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
• On the Theory of Transfer Learning: The Importance of Task Diversity
• Few-Shot Learning via Learning the Representation, Provably
• A Non-generative Framework and Convex Relaxations for Unsupervised Learning
• PAC-Bayesian Contrastive Unsupervised Representation Learning