A Discrete Hard EM Approach for Weakly Supervised Question Answering

Abstract

Many question answering (QA) tasks only provide weak supervision for how the answer should be computed. For example, TriviaQA answers are entities that can be mentioned multiple times in supporting documents, while DROP answers can be computed by deriving many different equations from numbers in the reference text. In this paper, we show it is possible to convert such tasks into discrete latent variable learning problems with a precomputed, task-specific set of possible "solutions" (e.g. different mentions or equations) that contains one correct option. We then develop a hard EM learning scheme that computes gradients relative to the most likely solution at each update. Despite its simplicity, we show that this approach significantly outperforms previous methods on six QA tasks, including absolute gains of 2--10%, and achieves the state-of-the-art on five of them. Using hard updates instead of maximizing marginal likelihood is key to these results as it encourages the model to find the one correct answer, which we show through detailed qualitative analysis.

Published as a conference paper at EMNLP 2019 (long). Code available at https://github.com/shmsw25/qa-hard-em

References in corpus (12)

• Bidirectional Attention Flow for Machine Comprehension
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
• A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization
• Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
• A BERT Baseline for the Natural Questions
• DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
• Learning to Generalize from Sparse and Underspecified Rewards
• X-SQL: reinforce schema representation with context
• Densely Connected Attention Propagation for Reading Comprehension
• Neural Program Synthesis with Priority Queue Training

Cited by in corpus (10)

• REALM: Retrieval-Augmented Language Model Pre-Training
• Semantic Models for the First-stage Retrieval: A Comprehensive Review
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
• Differentiable Reasoning over a Virtual Knowledge Base
• SeqGenSQL -- A Robust Sequence Generation Model for Structured Query Language
• SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval
• Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests
• Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study
• Frustratingly Hard Evidence Retrieval for QA Over Books
• Efficient Retrieval Optimized Multi-task Learning