CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers

Abstract

Dialogue state trackers have made significant progress on benchmark datasets, but their generalization capability to novel and realistic scenarios beyond the held-out conversations is less understood. We propose controllable counterfactuals (CoCo) to bridge this gap and evaluate dialogue state tracking (DST) models on novel scenarios, i.e., would the system successfully tackle the request if the user responded differently but still consistently with the dialogue flow? CoCo leverages turn-level belief states as counterfactual conditionals to produce novel conversation scenarios in two steps: (i) counterfactual goal generation at turn-level by dropping and adding slots followed by replacing slot values, (ii) counterfactual conversation generation that is conditioned on (i) and consistent with the dialogue flow. Evaluating state-of-the-art DST models on MultiWOZ dataset with CoCo-generated counterfactuals results in a significant performance drop of up to 30.8% (from 49.4% to 18.6%) in absolute joint goal accuracy. In comparison, widely used techniques like paraphrasing only affect the accuracy by at most 2%. Human evaluations show that COCO-generated conversations perfectly reflect the underlying user goal with more than 95% accuracy and are as human-like as the original conversations, further strengthening its reliability and promise to be adopted as part of the robustness evaluation of DST models.

ICLR 2021

References in corpus (11)

• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• A Simple Language Model for Task-Oriented Dialogue
• DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
• Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
• Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System
• Dialog State Tracking: A Neural Reading Comprehension Approach
• Improving Robustness of Task Oriented Dialog Systems
• TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking
• Neural Assistant: Joint Action Prediction, Response Generation, and Latent Knowledge Reasoning
• Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking
• MA-DST: Multi-Attention Based Scalable Dialog State Tracking

Cited by in corpus (5)

• KLUE: Korean Language Understanding Evaluation
• MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation
• Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking
• Zero-Shot Dialogue State Tracking via Cross-Task Transfer
• N-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking