How To Evaluate Your Dialogue System: Probe Tasks as an Alternative for Token-level Evaluation Metrics
arXiv:2008.10427
Abstract
Though generative dialogue modeling is widely seen as a language modeling task, the task demands an agent to have a complex natural language understanding of its input text to carry a meaningful interaction with an user. The automatic metrics used evaluate the quality of the generated text as a proxy to the holistic interaction of the agent. Such metrics were earlier shown to not correlate with the human judgement. In this work, we observe that human evaluation of dialogue agents can be inconclusive due to the lack of sufficient information for appropriate evaluation. The automatic metrics are deterministic yet shallow and human evaluation can be relevant yet inconclusive. To bridge this gap in evaluation, we propose designing a set of probing tasks to evaluate dialogue models. The hand-crafted tasks are aimed at quantitatively evaluating a generative dialogue model's understanding beyond the token-level evaluation on the generated text. The probing tasks are deterministic like automatic metrics and requires human judgement in their designing; benefiting from the best of both worlds. With experiments on probe tasks we observe that, unlike RNN based architectures, transformer model may not be learning to comprehend the input text despite its generated text having higher overlap with the target text.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Language Models are Few-Shot Learners
- A Neural Conversational Model
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Wizard of Wikipedia: Knowledge-Powered Conversational agents
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
- Deep Reinforcement Learning for Dialogue Generation
- Adversarial Learning for Neural Dialogue Generation
- Personalizing Dialogue Agents: I have a dog, do you have pets too?
- RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
- Towards Universal Paraphrastic Sentence Embeddings
- ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
- Neural Assistant: Joint Action Prediction, Response Generation, and Latent Knowledge Reasoning