SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems
arXiv:2110.06800 · doi:10.1609/aaai.v36i10.21341
Abstract
Zero/few-shot transfer to unseen services is a critical challenge in task-oriented dialogue research. The Schema-Guided Dialogue (SGD) dataset introduced a paradigm for enabling models to support any service in zero-shot through schemas, which describe service APIs to models in natural language. We explore the robustness of dialogue systems to linguistic variations in schemas by designing SGD-X - a benchmark extending SGD with semantically similar yet stylistically diverse variants for every schema. We observe that two top state tracking models fail to generalize well across schema variants, measured by joint goal accuracy and a novel metric for measuring schema sensitivity. Additionally, we present a simple model-agnostic data augmentation method to improve schema robustness.
arXiv commentAAAI 2022
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Synthetic and Natural Noise Both Break Neural Machine Translation
- Adversarial Examples for Evaluating Reading Comprehension Systems
- Schema-Guided Dialogue State Tracking Task at DSTC8
- STAR: A Schema-Guided Dialog Dataset for Transfer Learning
- An End-to-End Dialogue State Tracking System with Machine Reading Comprehension and Wide & Deep Classification
- Improving Robustness of Task Oriented Dialog Systems
- Fine-Tuning BERT for Schema-Guided Zero-Shot Dialogue State Tracking
- A Fast and Robust BERT-based Dialogue State Tracker for Schema-Guided Dialogue Dataset
- The SPPD System for Schema Guided Dialogue State Tracking Challenge
- Zero-shot Generalization in Dialog State Tracking through Generative Question Answering
- SGD-QA: Fast Schema-Guided Dialogue State Tracking for Unseen Services
- Robustness Testing of Language Understanding in Task-Oriented Dialog
- Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking