An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training
arXiv:2010.13116
Abstract
A class of recent semi-supervised learning (SSL) methods heavily rely on domain-specific data augmentations. In contrast, generative SSL methods involve unsupervised learning based on generative models by either joint-training or pre-training, and are more appealing from the perspective of being domain-agnostic, since they do not inherently require data augmentations. Joint-training estimates the joint distribution of observations and labels, while pre-training is taken over observations only. Recently, energy-based models (EBMs) have achieved promising results for generative modeling. Joint-training via EBMs for SSL has been explored with encouraging results across different data modalities. In this paper, we make two contributions. First, we explore pre-training via EBMs for SSL and compare it to joint-training. Second, a suite of experiments are conducted over domains of image classification and natural language labeling to give a realistic whole picture of the performances of EBM based SSL methods. It is found that joint-training EBMs outperform pre-training EBMs marginally but nearly consistently.
References in corpus (8)
- A Simple Framework for Contrastive Learning of Visual Representations
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Semi-Supervised Learning with Ladder Networks
- Rethinking Pre-training and Self-training
- Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
- A Theory of Generative ConvNet
- Implicit Generation and Generalization in Energy-Based Models