Semi-supervised Sequential Generative Models

Abstract

We introduce a novel objective for training deep generative time-series models with discrete latent variables for which supervision is only sparsely available. This instance of semi-supervised learning is challenging for existing methods, because the exponential number of possible discrete latent configurations results in high variance gradient estimators. We first overcome this problem by extending the standard semi-supervised generative modeling objective with reweighted wake-sleep. However, we find that this approach still suffers when the frequency of available labels varies between training sequences. Finally, we introduce a unified objective inspired by teacher-forcing and show that this approach is robust to variable length supervision. We call the resulting method caffeinated wake-sleep (CWS) to emphasize its additional dependence on real data. We demonstrate its effectiveness with experiments on MNIST, handwriting, and fruit fly trajectory data.

Accepted to Uncertainty in Artificial Intelligence 2020

References in corpus (13)

• Categorical Reparameterization with Gumbel-Softmax
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Semi-Supervised Learning with Deep Generative Models
• A Recurrent Latent Variable Model for Sequential Data
• Emergence of grid-like representations by training recurrent neural networks to perform spatial localization
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Variational inference for Monte Carlo objectives
• Learning Disentangled Representations with Semi-Supervised Deep Generative Models
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Reweighted Wake-Sleep
• Learning recurrent representations for hierarchical behavior modeling
• Learning Wake-Sleep Recurrent Attention Models