Know What You Don't Know: Unanswerable Questions for SQuAD

Abstract

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuAD 2.0, the latest version of the Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0.

arXiv commentACL 2018

Cited by in corpus (63)

• LoRA: Low-Rank Adaptation of Large Language Models
• Pathologies of Neural Models Make Interpretations Difficult
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• Transforming Question Answering Datasets Into Natural Language Inference Datasets
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• A Survey on Text Classification: From Shallow to Deep Learning
• CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
• Testing Robustness Against Unforeseen Adversaries
• Structured Pruning of a BERT-based Question Answering Model
• QuAC : Question Answering in Context
• BinaryBERT: Pushing the Limit of BERT Quantization
• Pay Attention to MLPs
• Explicit Contextual Semantics for Text Comprehension
• SG-Net: Syntax-Guided Machine Reading Comprehension
• Neural Generation Meets Real People: Towards Emotionally Engaging Mixed-Initiative Conversations
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• A Comparative Study of Transformer-Based Language Models on Extractive Question Answering
• Read + Verify: Machine Reading Comprehension with Unanswerable Questions
• FQuAD: French Question Answering Dataset
• What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
• BERT Based Multilingual Machine Comprehension in English and Hindi
• What Makes Reading Comprehension Questions Easier?
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
• Towards Understanding and Answering Multi-Sentence Recommendation Questions on Tourism
• ExpMRC: Explainability Evaluation for Machine Reading Comprehension
• Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models
• Incorporating Structured Commonsense Knowledge in Story Completion
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting
• A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets
• MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims
• A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering
• Code to Comment "Translation": Data, Metrics, Baselining & Evaluation
• HEAD-QA: A Healthcare Dataset for Complex Reasoning
• A Survey On Neural Word Embeddings
• Hidden Backdoors in Human-Centric Language Models
• Probing Emergent Semantics in Predictive Agents via Question Answering
• Contextual Aware Joint Probability Model Towards Question Answering System
• TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions
• ReCO: A Large Scale Chinese Reading Comprehension Dataset on Opinion
• Instance-based Transfer Learning for Multilingual Deep Retrieval
• Knowledge Based Machine Reading Comprehension
• Answering Any-hop Open-domain Questions with Iterative Document Reranking
• Commonsense knowledge adversarial dataset that challenges ELECTRA
• QUACKIE: A NLP Classification Task With Ground Truth Explanations
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
• Weakly supervised one-stage vision and language disease detection using large scale pneumonia and pneumothorax studies
• Exploring Neural Net Augmentation to BERT for Question Answering on SQUAD 2.0
• Situation and Behavior Understanding by Trope Detection on Films
• BERTnesia: Investigating the capture and forgetting of knowledge in BERT
• Recent Advances in Automated Question Answering In Biomedical Domain
• Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study
• Inquisitive Question Generation for High Level Text Comprehension
• Fine-tuning Multi-hop Question Answering with Hierarchical Graph Network
• LV-BERT: Exploiting Layer Variety for BERT
• Sentence-level Online Handwritten Chinese Character Recognition
• Action based Network for Conversation Question Reformulation
• Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching
• Towards Explainable Fact Checking
• Indian Legal NLP Benchmarks : A Survey
• Outbound Translation User Interface Ptakopet: A Pilot Study
• Empirically Verifying Hypotheses Using Reinforcement Learning