Improving Variational Encoder-Decoders in Dialogue Generation

Abstract

Variational encoder-decoders (VEDs) have shown promising results in dialogue generation. However, the latent variable distributions are usually approximated by a much simpler model than the powerful RNN structure used for encoding and decoding, yielding the KL-vanishing problem and inconsistent training objective. In this paper, we separate the training step into two phases: The first phase learns to autoencode discrete texts into continuous embeddings, from which the second phase learns to generalize latent representations by reconstructing the encoded embedding. In this case, latent variables are sampled by transforming Gaussian noise through multi-layer perceptrons and are trained with a separate VED model, which has the potential of realizing a much more flexible distribution. We compare our model with current popular models and the experiment demonstrates substantial improvement in both metric-based and human evaluations.

Accepted by AAAI2018

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• A Neural Conversational Model
• Sequence Transduction with Recurrent Neural Networks
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Variational Lossy Autoencoder
• Data Noising as Smoothing in Neural Network Language Models
• A Hybrid Convolutional Variational Autoencoder for Text Generation

Cited by in corpus (5)

• DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder
• mu-Forcing: Training Variational Recurrent Autoencoders for Text Generation
• Better Conversations by Modeling,Filtering,and Optimizing for Coherence and Diversity
• Aiming to Know You Better Perhaps Makes Me a More Engaging Dialogue Partner
• Posterior-GAN: Towards Informative and Coherent Response Generation with Posterior Generative Adversarial Network