Text Modeling with Syntax-Aware Variational Autoencoders

Abstract

Syntactic information contains structures and rules about how text sentences are arranged. Incorporating syntax into text modeling methods can potentially benefit both representation learning and generation. Variational autoencoders (VAEs) are deep generative models that provide a probabilistic way to describe observations in the latent space. When applied to text data, the latent representations are often unstructured. We propose syntax-aware variational autoencoders (SAVAEs) that dedicate a subspace in the latent dimensions dubbed syntactic latent to represent syntactic structures of sentences. SAVAEs are trained to infer syntactic latent from either text inputs or parsed syntax results as well as reconstruct original text with inferred latent variables. Experiments show that SAVAEs are able to achieve lower reconstruction loss on four different data sets. Furthermore, they are capable of generating examples with modified target syntax.

References in corpus (11)

• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Semi-Supervised Learning with Deep Generative Models
• Variational Inference with Normalizing Flows
• Style Transfer from Non-Parallel Text by Cross-Alignment
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Syntax-Directed Variational Autoencoder for Structured Data
• Variational Reasoning for Question Answering with Knowledge Graph
• Autoencoding Variational Inference For Topic Models
• Variational Knowledge Graph Reasoning
• Dirichlet Variational Autoencoder for Text Modeling