Semi-supervised Learning with Contrastive Predicative Coding

Abstract

Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, many of them have thus far been either inflexible, inefficient or non-scalable. This paper explores recently developed contrastive predictive coding technique to improve discriminative power of deep learning models when a large portion of labels are absent. Two models, cpc-SSL and a class conditional variant~(ccpc-SSL) are presented. They effectively exploit the unlabeled data by extracting shared information between different parts of the (high-dimensional) data. The proposed approaches are inductive, and scale well to very large datasets like ImageNet, making them good candidates in real-world large scale applications.

6 pages, 4 figures, conference

References in corpus (12)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Categorical Reparameterization with Gumbel-Softmax
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Semi-Supervised Learning with Ladder Networks
• Learning with Pseudo-Ensembles
• Triple Generative Adversarial Nets
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• Learning by Association - A versatile semi-supervised training method for neural networks
• Semi-Supervised Learning via Compact Latent Space Clustering
• Max-margin Deep Generative Models