Natural Language Generation with Neural Variational Models
arXiv:1808.09012
Abstract
In this thesis, we explore the use of deep neural networks for generation of natural language. Specifically, we implement two sequence-to-sequence neural variational models - variational autoencoders (VAE) and variational encoder-decoders (VED). VAEs for text generation are difficult to train due to issues associated with the Kullback-Leibler (KL) divergence term of the loss function vanishing to zero. We successfully train VAEs by implementing optimization heuristics such as KL weight annealing and word dropout. We also demonstrate the effectiveness of this continuous latent space through experiments such as random sampling, linear interpolation and sampling from the neighborhood of the input. We argue that if VAEs are not designed appropriately, it may lead to bypassing connections which results in the latent space being ignored during training. We show experimentally with the example of decoder hidden state initialization that such bypassing connections degrade the VAE into a deterministic model, thereby reducing the diversity of generated sentences. We discover that the traditional attention mechanism used in sequence-to-sequence VED models serves as a bypassing connection, thereby deteriorating the model's latent space. In order to circumvent this issue, we propose the variational attention mechanism where the attention context vector is modeled as a random variable that can be sampled from a distribution. We show empirically using automatic evaluation metrics, namely entropy and distinct measures, that our variational attention model generates more diverse output sentences than the deterministic attention model. A qualitative analysis with human evaluation study proves that our model simultaneously produces sentences that are of high quality and equally fluent as the ones generated by the deterministic attention counterpart.
arXiv commentMasters Thesis, University of Waterloo
References in corpus (36)
- Adam: A Method for Stochastic Optimization
- Deep Learning in Neural Networks: An Overview
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- On the difficulty of training Recurrent Neural Networks
- End to End Learning for Self-Driving Cars
- NLTK: The Natural Language Toolkit
- A Neural Conversational Model
- Tutorial on Variational Autoencoders
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Deep Convolutional Inverse Graphics Network
- Reasoning about Entailment with Neural Attention
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- A large annotated corpus for learning natural language inference
- Deep Reinforcement Learning for Dialogue Generation
- LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection
- Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
- Deep Generative Stochastic Networks Trainable by Backprop
- Adversarial Learning for Neural Dialogue Generation
- A Hierarchical Neural Autoencoder for Paragraphs and Documents
- Sequence to Sequence -- Video to Text
- Long Short-Term Memory-Networks for Machine Reading
- A Persona-Based Neural Conversation Model
- Sequence-to-Sequence Learning as Beam-Search Optimization
- Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path
- Learning to Ask: Neural Question Generation for Reading Comprehension
- Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
- A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
- Learning End-to-End Goal-Oriented Dialog
- Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion
- Neural Generative Question Answering
- Variational Neural Machine Translation
- Latent Variable Dialogue Models and their Diversity
- An Attentional Neural Conversation Model with Improved Specificity
- Variational Attention for Sequence-to-Sequence Models
- Question Generation from a Knowledge Base with Web Exploration