Probing Neural Dialog Models for Conversational Understanding
arXiv:2006.08331 · doi:10.18653/v1/2020.nlp4convai-1.15
Abstract
The predominant approach to open-domain dialog generation relies on end-to-end training of neural models on chat datasets. However, this approach provides little insight as to what these models learn (or do not learn) about engaging in dialog. In this study, we analyze the internal representations learned by neural open-domain dialog systems and evaluate the quality of these representations for learning basic conversational skills. Our results suggest that standard open-domain dialog systems struggle with answering questions, inferring contradiction, and determining the topic of conversation, among other tasks. We also find that the dyadic, turn-taking nature of dialog is not fully leveraged by these models. By exploring these limitations, we highlight the need for additional research into architectures and training methods that can better capture high-level information about dialog.
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- Universal Sentence Encoder
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Skip-Thought Vectors
- DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- Deep Reinforcement Learning for Dialogue Generation
- Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
- Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
- What do Neural Machine Translation Models Learn about Morphology?
- Adversarial Learning for Neural Dialogue Generation
- Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
- Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
- Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
- Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
- Towards Explainable and Controllable Open Domain Dialogue Generation with Dialogue Acts
- Dialogue Natural Language Inference
- ScenarioSA: A Large Scale Conversational Database for Interactive Sentiment Analysis