$k$Folden: $k$-Fold Ensemble for Out-Of-Distribution Detection

Abstract

Out-of-Distribution (OOD) detection is an important problem in natural language processing (NLP). In this work, we propose a simple yet effective framework $k$Folden, which mimics the behaviors of OOD detection during training without the use of any external data. For a task with $k$ training labels, $k$Folden induces $k$ sub-models, each of which is trained on a subset with $k-1$ categories with the left category masked unknown to the sub-model. Exposing an unknown label to the sub-model during training, the model is encouraged to learn to equally attribute the probability to the seen $k-1$ labels for the unknown label, enabling this framework to simultaneously resolve in- and out-distribution examples in a natural way via OOD simulations. Taking text classification as an archetype, we develop benchmarks for OOD detection using existing text classification datasets. By conducting comprehensive comparisons and analyses on the developed benchmarks, we demonstrate the superiority of $k$Folden against current methods in terms of improving OOD detection performances while maintaining improved in-domain classification accuracy. The code and datasets can be found at: \url{https://github.com/ShannonAI/kfolden-ood-detection}.

arXiv commentTo appear at EMNLP 2021

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Character-level Convolutional Networks for Text Classification
• Theoretical Models of Learning to Learn
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• On First-Order Meta-Learning Algorithms
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Deep Anomaly Detection with Outlier Exposure
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
• Outlier Exposure with Confidence Control for Out-of-Distribution Detection
• BertGCN: Transductive Text Classification by Combining GCN and BERT
• Uncertainty-aware Self-training for Text Classification with Few Labels
• Calibration of Encoder Decoder Models for Neural Machine Translation
• Meta-Learning for Low-Resource Neural Machine Translation
• Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing
• Selective Question Answering under Domain Shift
• Learning to Customize Model Structures for Few-shot Dialogue Generation Tasks
• Learning to Learn and Predict: A Meta-Learning Approach for Multi-Label Classification