What If We Simply Swap the Two Text Fragments? A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Tasks
arXiv:1809.02719
Abstract
Nature language inference (NLI) task is a predictive task of determining the inference relationship of a pair of natural language sentences. With the increasing popularity of NLI, many state-of-the-art predictive models have been proposed with impressive performances. However, several works have noticed the statistical irregularities in the collected NLI data set that may result in an over-estimated performance of these models and proposed remedies. In this paper, we further investigate the statistical irregularities, what we refer as confounding factors, of the NLI data sets. With the belief that some NLI labels should preserve under swapping operations, we propose a simple yet effective way (swapping the two text fragments) of evaluating the NLI predictive models that naturally mitigate the observed problems. Further, we continue to train the predictive models with our swapping manner and propose to use the deviation of the model's evaluation performances under different percentages of training text fragments to be swapped to describe the robustness of a predictive model. Our evaluation metrics leads to some interesting understandings of recent published NLI methods. Finally, we also apply the swapping operation on NLI models to see the effectiveness of this straightforward method in mitigating the confounding factor problems in training generic sentence embeddings for other NLP transfer tasks.
arXiv comment8 pages, to appear at AAAI 19
References in corpus (19)
- Confounding variables can degrade generalization performance of radiological deep learning models
- Enhanced LSTM for Natural Language Inference
- Reasoning about Entailment with Neural Attention
- A large annotated corpus for learning natural language inference
- Neural Natural Language Inference Models Enhanced with External Knowledge
- Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
- Long Short-Term Memory-Networks for Machine Reading
- Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
- Order-Embeddings of Images and Language
- Learning Natural Language Inference with LSTM
- Stochastic Answer Networks for Natural Language Inference
- Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge
- Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information
- Shortcut-Stacked Sentence Encoders for Multi-Domain Inference
- A Fast Unified Model for Parsing and Sentence Understanding
- Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
- Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference
- Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis
- Modelling Interaction of Sentence Pair with coupled-LSTMs