BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Abstract

In this paper we study yes/no questions that are naturally occurring --- meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.

In NAACL 2019

References in corpus (2)

• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

Cited by in corpus (25)

• Language Models are Few-Shot Learners
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Entailment as Few-Shot Learner
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• KLUE: Korean Language Understanding Evaluation
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Brain
• Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality
• Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks
• CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge
• Sequence tagging for biomedical extractive question answering
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• MultiModalQA: Complex Question Answering over Text, Tables and Images
• Unified Question Generation with Continual Lifelong Learning
• DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
• DaNetQA: a yes/no Question Answering Dataset for the Russian Language
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
• Natural Language QA Approaches using Reasoning with External Knowledge
• FLEX: Unifying Evaluation for Few-Shot NLP
• Evaluating and Characterizing Human Rationales
• Science Checker: Extractive-Boolean Question Answering For Scientific Fact Checking
• Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
• Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
• Towards Explainable Fact Checking
• Zero-shot Task Transfer for Invoice Extraction via Class-aware QA Ensemble