TabFact: A Large-scale Dataset for Table-based Fact Verification

Abstract

The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains under-explored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far behind human performance. We also perform a comprehensive analysis to demonstrate great future opportunities. The data and code of the dataset are provided in \url{https://github.com/wenhuchen/Table-Fact-Checking}.

arXiv commentAccepted to ICLR 2020, 17 pages, 15 figures. Main paper has 9 pages

References in corpus (11)

• Enhanced LSTM for Natural Language Inference
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• A large annotated corpus for learning natural language inference
• Language to Logical Form with Neural Attention
• End-to-End Differentiable Proving
• Compositional Semantic Parsing on Semi-Structured Tables
• A Syntactic Neural Model for General-Purpose Code Generation
• Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
• UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification

Cited by in corpus (21)

• TAPAS: Weakly Supervised Table Parsing via Pre-training
• TAPEX: Table Pre-training via Learning a Neural SQL Executor
• The State of Human-centered NLP Technology for Fact-checking
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
• GitTables: A Large-Scale Corpus of Relational Tables
• Improving Text-to-SQL with Schema Dependency Learning
• On Grounded Planning for Embodied Tasks with Language Models
• Profile Consistency Identification for Open-domain Dialogue Agents
• MultiModalQA: Complex Question Answering over Text, Tables and Images
• HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
• Stage-wise Fine-tuning for Graph-to-Text Generation
• Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
• Logic2Text: High-Fidelity Natural Language Generation from Logical Forms
• BERT Meets Relational DB: Contextual Representations of Relational Databases
• Open Question Answering over Tables and Text
• WhatTheWikiFact: Fact-Checking Claims Against Wikipedia
• CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
• Leveraging Declarative Knowledge in Text and First-Order Logic for Fine-Grained Propaganda Detection
• Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning
• Program Enhanced Fact Verification with Verbalization and Graph Attention Network
• Sattiy at SemEval-2021 Task 9: An Ensemble Solution for Statement Verification and Evidence Finding with Tables