A Tutorial on Deep Latent Variable Models of Natural Language

Abstract

There has been much recent, exciting work on combining the complementary strengths of latent variable models and deep learning. Latent variable modeling makes it easy to explicitly specify model constraints through conditional independence properties, while deep learning makes it possible to parameterize these conditional likelihoods with powerful function approximators. While these "deep latent variable" models provide a rich, flexible framework for modeling many real-world phenomena, difficulties exist: deep parameterizations of conditional likelihoods usually make posterior inference intractable, and latent variable objectives often complicate backpropagation by introducing points of non-differentiability. This tutorial explores these issues in depth through the lens of variational inference.

arXiv commentEMNLP 2018 Tutorial

References in corpus (103)

• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Categorical Reparameterization with Gumbel-Softmax
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Recurrent Neural Network Regularization
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Improved Techniques for Training GANs
• Variational Graph Auto-Encoders
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Density estimation using Real NVP
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
• Disentangling by Factorising
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• Pyro: Deep Universal Probabilistic Programming
• Neural Ordinary Differential Equations
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• Regularizing and Optimizing LSTM Language Models
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• MINE: Mutual Information Neural Estimation
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• InfoVAE: Information Maximizing Variational Autoencoders
• MADE: Masked Autoencoder for Distribution Estimation
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Adversarial Autoencoders
• Variational Lossy Autoencoder
• Importance Weighted Autoencoders
• Syntax-Directed Variational Autoencoder for Structured Data
• Composing graphical models with neural networks for structured representations and fast inference
• Discovering Discrete Latent Topics with Neural Variational Inference
• Adversarial Learning for Neural Dialogue Generation
• Masked Autoregressive Flow for Density Estimation
• Learning in Implicit Generative Models
• Glow: Generative Flow with Invertible 1x1 Convolutions
• Improving Variational Inference with Inverse Autoregressive Flow
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Long Text Generation via Adversarial Training with Leaked Information
• Sequential Neural Models with Stochastic Layers
• Syntactic Topic Models
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Toward Controlled Generation of Text
• Variational inference for Monte Carlo objectives
• Learning Factored Representations in a Deep Mixture of Experts
• TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
• Implicit Reparameterization Gradients
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder
• Grammar Variational Autoencoder
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Sylvester Normalizing Flows for Variational Inference
• Segmental Recurrent Neural Networks
• Hyperspherical Variational Auto-Encoders
• Adversarial Generation of Natural Language
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training
• Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
• Improving Variational Auto-Encoders using Householder Flow
• Latent Alignment and Variational Attention
• Inference Suboptimality in Variational Autoencoders
• Language GANs Falling Short
• On Accurate Evaluation of GANs for Language Generation
• Auto-Encoding Sequential Monte Carlo
• Topic Compositional Neural Language Model
• Theory and Experiments on Vector Quantized Autoencoders
• Reinterpreting Importance-Weighted Autoencoders
• A Hybrid Convolutional Variational Autoencoder for Text Generation
• Stick-Breaking Variational Autoencoders
• Reducing Reparameterization Gradient Variance
• Language as a Latent Variable: Discrete Generative Models for Sentence Compression
• Learning Hierarchical Features from Generative Models
• Avoiding Latent Variable Collapse With Generative Skip Models
• Iterative Amortized Inference
• Deconvolutional Latent-Variable Model for Text Sequence Matching
• Generating Sentences by Editing Prototypes
• Eval all, trust a few, do wrong to none: Comparing sentence generation models
• Spherical Latent Spaces for Stable Variational Autoencoders
• Auto-Encoding Total Correlation Explanation
• On the challenges of learning with inference networks on sparse, high-dimensional data
• The Generalized Reparameterization Gradient
• Tighter Variational Bounds are Not Necessarily Better
• Advances in Variational Inference
• Evaluating Text GANs as Language Models
• Likelihood-free inference via classification
• On parameter estimation with the Wasserstein distance
• Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow
• The Neural Noisy Channel
• Learnable Explicit Density for Continuous Latent Space and Variational Inference
• Unsupervised POS Induction with Word Embeddings
• A Hierarchical Latent Structure for Variational Conversation Modeling
• Pushing the bounds of dropout
• Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives
• Nonparametric Inference for Auto-Encoding Variational Bayes
• Learning Neural Templates for Text Generation
• Unsupervised Neural Hidden Markov Models
• CRF Autoencoder for Unsupervised Dependency Parsing
• Learning to Discover, Ground and Use Words with Segmental Neural Language Models
• Hard Non-Monotonic Attention for Character-Level Transduction

Cited by in corpus (15)

• Evaluation of Text Generation: A Survey
• Fast Structured Decoding for Sequence Models
• Neural Language Generation: Formulation, Methods, and Evaluation
• Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
• Music FaderNets: Controllable Music Generation Based On High-Level Features via Low-Level Feature Modelling
• Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity
• Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence
• Learning to Predict Explainable Plots for Neural Story Generation
• Exploring Bayesian Deep Learning for Urgent Instructor Intervention Need in MOOC Forums
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Amortized Bethe Free Energy Minimization for Learning MRFs
• Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers
• Language as a Latent Sequence: deep latent variable models for semi-supervised paraphrase generation
• BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition
• To be Closer: Learning to Link up Aspects with Opinions