Adversarial Evaluation of Dialogue Models

Abstract

The recent application of RNN encoder-decoder models has resulted in substantial progress in fully data-driven dialogue systems, but evaluation remains a challenge. An adversarial loss could be a way to directly evaluate the extent to which generated dialogue responses sound like they came from a human. This could reduce the need for human evaluation, while more directly evaluating on a generative task. In this work, we investigate this idea by training an RNN to discriminate a dialogue model's samples from human-generated samples. Although we find some evidence this setup could be viable, we also note that many issues remain in its practical application. We discuss both aspects and conclude that future work is warranted.

References in corpus (5)

• Improved Techniques for Training GANs
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Deep Reinforcement Learning for Dialogue Generation
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Smart Reply: Automated Response Suggestion for Email

Cited by in corpus (34)

• Adversarial Learning for Neural Dialogue Generation
• Natural Language Processing Advancements By Deep Learning: A Survey
• Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
• Challenges in Data-to-Document Generation
• Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge
• Unifying Human and Statistical Evaluation for Natural Language Generation
• Neural Language Generation: Formulation, Methods, and Evaluation
• Data Distillation for Controlling Specificity in Dialogue Generation
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• Designing for Health Chatbots
• Neural Net Models for Open-Domain Discourse Coherence
• PONE: A Novel Automatic Evaluation Metric for Open-Domain Generative Dialogue Systems
• Learning to Evaluate Image Captioning
• Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• Non-Autoregressive Neural Dialogue Generation
• A Survey of Document Grounded Dialogue Systems (DGDS)
• Logical Natural Language Generation from Open-Domain Tables
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• A Measure for Dialog Complexity and its Application in Streamlining Service Operations
• Improved Sentiment Detection via Label Transfer from Monolingual to Synthetic Code-Switched Text
• Teaching Machines to Converse
• How to Evaluate the Next System: Automatic Dialogue Evaluation from the Perspective of Continual Learning
• Communication-based Evaluation for Natural Language Generation
• Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
• Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models
• OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics
• UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation
• Fine-grained Sentiment Controlled Text Generation
• How to Evaluate Your Dialogue Models: A Review of Approaches
• Coupled Generative Adversarial Network for Continuous Fine-grained Action Segmentation
• On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems
• Seq2Seq Mimic Games: A Signaling Perspective
• Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems