BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
arXiv:1905.10044
Abstract
In this paper we study yes/no questions that are naturally occurring --- meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.
arXiv commentIn NAACL 2019
References in corpus (9)
- Bidirectional Attention Flow for Machine Comprehension
- Enhanced LSTM for Natural Language Inference
- VQA: Visual Question Answering
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- Transforming Question Answering Datasets Into Natural Language Inference Datasets
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
- Read + Verify: Machine Reading Comprehension with Unanswerable Questions
Cited by in corpus (27)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Language Models are Few-Shot Learners
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Entailment as Few-Shot Learner
- Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
- KLUE: Korean Language Understanding Evaluation
- Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
- Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Brain
- Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality
- Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks
- CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge
- Sequence tagging for biomedical extractive question answering
- LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
- MultiModalQA: Complex Question Answering over Text, Tables and Images
- Unified Question Generation with Continual Lifelong Learning
- DaNetQA: a yes/no Question Answering Dataset for the Russian Language
- DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
- ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
- Natural Language QA Approaches using Reasoning with External Knowledge
- FLEX: Unifying Evaluation for Few-Shot NLP
- Evaluating and Characterizing Human Rationales
- Science Checker: Extractive-Boolean Question Answering For Scientific Fact Checking
- Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
- Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
- Zero-shot Task Transfer for Invoice Extraction via Class-aware QA Ensemble
- Towards Explainable Fact Checking