MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation

Abstract

The MultiWOZ 2.0 dataset has greatly stimulated the research of task-oriented dialogue systems. However, its state annotations contain substantial noise, which hinders a proper evaluation of model performance. To address this issue, massive efforts were devoted to correcting the annotations. Three improved versions (i.e., MultiWOZ 2.1-2.3) have then been released. Nonetheless, there are still plenty of incorrect and inconsistent annotations. This work introduces MultiWOZ 2.4, which refines the annotations in the validation set and test set of MultiWOZ 2.1. The annotations in the training set remain unchanged (same as MultiWOZ 2.1) to elicit robust and noise-resilient model training. We benchmark eight state-of-the-art dialogue state tracking models on MultiWOZ 2.4. All of them demonstrate much higher performance than on MultiWOZ 2.1.

Accepted to SIGDIAL 2022 (https://2022.sigdial.org/)

References in corpus (5)

• A Survey of Label-noise Representation Learning: Past, Present and Future
• CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers
• MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines
• MultiWOZ 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation
• Slot Self-Attentive Dialogue State Tracking

Cited by in corpus (2)

• POSSCORE: A Simple Yet Effective Evaluation of Conversational Search with Part of Speech Labelling
• Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances