Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation
arXiv:2104.07555
Abstract
QuestEval is a reference-less metric used in text-to-text tasks, that compares the generated summaries directly to the source text, by automatically asking and answering questions. Its adaptation to Data-to-Text tasks is not straightforward, as it requires multimodal Question Generation and Answering systems on the considered tasks, which are seldom available. To this purpose, we propose a method to build synthetic multimodal corpora enabling to train multimodal components for a data-QuestEval metric. The resulting metric is reference-less and multimodal; it obtains state-of-the-art correlations with human judgment on the WebNLG and WikiBio benchmarks. We make data-QuestEval's code and models available for reproducibility purpose, as part of the QuestEval project.
arXiv commentAccepted at EMNLP 2021
References in corpus (7)
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- BERTScore: Evaluating Text Generation with BERT
- TAPAS: Weakly Supervised Table Parsing via Pre-training
- Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
- Text-to-Text Pre-Training for Data-to-Text Tasks
- QuestEval: Summarization Asks for Fact-based Evaluation
- FeTaQA: Free-form Table Question Answering