Improving Robustness of Task Oriented Dialog Systems

Abstract

Task oriented language understanding in dialog systems is often modeled using intents (task of a query) and slots (parameters for that task). Intent detection and slot tagging are, in turn, modeled using sentence classification and word tagging techniques respectively. Similar to adversarial attack problems with computer vision models discussed in existing literature, these intent-slot tagging models are often over-sensitive to small variations in input -- predicting different and often incorrect labels when small changes are made to a query, thus reducing their accuracy and reliability. However, evaluating a model's robustness to these changes is harder for language since words are discrete and an automated change (e.g. adding `noise') to a query sometimes changes the meaning and thus labels of a query. In this paper, we first describe how to create an adversarial test set to measure the robustness of these models. Furthermore, we introduce and adapt adversarial training methods as well as data augmentation using back-translation to mitigate these issues. Our experiments show that both techniques improve the robustness of the system substantially and can be combined to yield the best results.

References in corpus (6)

• Sequence to Sequence Learning with Neural Networks
• Unsupervised Data Augmentation for Consistency Training
• Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
• Semi-supervised Sequence Learning
• Unrestricted Adversarial Examples
• Efficient Augmentation via Data Subsampling

Cited by in corpus (9)

• CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers
• SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems
• Neural model robustness for skill routing in large-scale conversational AI systems: A design choice exploration
• AuGPT: Auxiliary Tasks and Data Augmentation for End-To-End Dialogue with Pre-Trained Language Models
• Data Augmentation for Training Dialog Models Robust to Speech Recognition Errors
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding
• On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise
• Semantic Representation for Dialogue Modeling
• TIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling