Grammar Variational Autoencoder

Abstract

Deep generative models have been wildly successful at learning coherent latent representations for continuous data such as video and audio. However, generative modeling of discrete data such as arithmetic expressions and molecular structures still poses significant challenges. Crucially, state-of-the-art methods often produce outputs that are not valid. We make the key observation that frequently, discrete data can be represented as a parse tree from a context-free grammar. We propose a variational autoencoder which encodes and decodes directly to and from these parse trees, ensuring the generated outputs are always valid. Surprisingly, we show that not only does our model more often generate valid outputs, it also learns a more coherent latent space in which nearby points decode to similar discrete outputs. We demonstrate the effectiveness of our learned models by showing their improved performance in Bayesian optimization for symbolic regression and molecular synthesis.

References in corpus (8)

• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• A Convolutional Neural Network for Modelling Sentences
• GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
• TerpreT: A Probabilistic Programming Language for Program Induction
• Generating Focussed Molecule Libraries for Drug Discovery with Recurrent Neural Networks
• Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control

Cited by in corpus (40)

• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Deep learning for molecular design - a review of the state of the art
• Learning Deep Generative Models of Graphs
• Autonomous discovery in the chemical sciences part II: Outlook
• ChemTS: An Efficient Python Library for de novo Molecular Generation
• Syntax-Directed Variational Autoencoder for Structured Data
• Conditional molecular design with deep generative models
• Learning Multimodal Graph-to-Graph Translation for Molecular Optimization
• Hierarchical Generation of Molecular Graphs using Structural Motifs
• Generative Models for Automatic Chemical Design
• Tree-to-tree Neural Networks for Program Translation
• Deeply learning molecular structure-property relationships using attention- and gate-augmented graph convolutional network
• Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories
• All SMILES Variational Autoencoder
• Simple, Distributed, and Accelerated Probabilistic Programming
• Latent Molecular Optimization for Targeted Therapeutic Design
• Simulated annealing for optimization of graphs and sequences
• A Tutorial on Deep Latent Variable Models of Natural Language
• Hierarchical Graph-to-Graph Translation for Molecules
• Generating equilibrium molecules with deep neural networks
• Graph Convolution: A High-Order and Adaptive Approach
• Neural Attribute Machines for Program Generation
• PepCVAE: Semi-Supervised Targeted Design of Antimicrobial Peptide Sequences
• Variational Autoencoder for Anti-Cancer Drug Response Prediction
• Neural-Guided Symbolic Regression with Asymptotic Constraints
• Mask-GVAE: Blind Denoising Graphs via Partition
• Direct Optimization through $\arg \max$ for Discrete Variational Auto-Encoder
• "Found in Translation": Predicting Outcomes of Complex Organic Chemistry Reactions using Neural Sequence-to-Sequence Models
• Deep Denerative Models for Drug Design and Response
• Improving Molecular Design by Stochastic Iterative Target Augmentation
• Physics-Constrained Predictive Molecular Latent Space Discovery with Graph Scattering Variational Autoencoder
• Goal-directed Generation of Discrete Structures with Conditional Generative Models
• Re-balancing Variational Autoencoder Loss for Molecule Sequence Generation
• ast2vec: Utilizing Recursive Neural Encodings of Python Programs
• Semi-Supervised Hierarchical Drug Embedding in Hyperbolic Space
• Benchmarking Deep Graph Generative Models for Optimizing New Drug Molecules for COVID-19
• A reproducibility study of "Augmenting Genetic Algorithms with Deep Neural Networks for Exploring the Chemical Space"
• Neighbor Embedding Variational Autoencoder
• Generate Novel Molecules With Target Properties Using Conditional Generative Models
• Encoded Prior Sliced Wasserstein AutoEncoder for learning latent manifold representations