Out-of-domain Detection for Natural Language Understanding in Dialog Systems
arXiv:1909.03862
Abstract
Natural Language Understanding (NLU) is a vital component of dialogue systems, and its ability to detect Out-of-Domain (OOD) inputs is critical in practical applications, since the acceptance of the OOD input that is unsupported by the current system may lead to catastrophic failure. However, most existing OOD detection methods rely heavily on manually labeled OOD samples and cannot take full advantage of unlabeled data. This limits the feasibility of these models in practical applications. In this paper, we propose a novel model to generate high-quality pseudo OOD samples that are akin to IN-Domain (IND) input utterances, and thereby improves the performance of OOD detection. To this end, an autoencoder is trained to map an input utterance into a latent code. and the codes of IND and OOD samples are trained to be indistinguishable by utilizing a generative adversarial network. To provide more supervision signals, an auxiliary classifier is introduced to regularize the generated OOD samples to have indistinguishable intent labels. Experiments show that these pseudo OOD samples generated by our model can be used to effectively improve OOD detection in NLU. Besides, we also demonstrate that the effectiveness of these pseudo OOD data can be further improved by efficiently utilizing unlabeled data.
arXiv commentAccepted by TASLP, Code available at https://github.com/silverriver/OOD4NLU
References in corpus (15)
- On Calibration of Modern Neural Networks
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Convolutional Neural Networks for Sentence Classification
- Recent Advances in Open Set Recognition: A Survey
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- One-Class Classification: Taxonomy of Study and Review of Techniques
- Deep Anomaly Detection with Outlier Exposure
- Deep Anomaly Detection Using Geometric Transformations
- Generative Probabilistic Novelty Detection with Adversarial Autoencoders
- Statistical Analysis of Nearest Neighbor Methods for Anomaly Detection
- Neural Sentence Embedding using Only In-domain Sentences for Out-of-domain Sentence Detection in Dialog Systems
- Adversarially Regularized Autoencoders
- Novelty Detection with GAN
- A Less Biased Evaluation of Out-of-distribution Sample Detectors
- Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog