83 citations · 111 across the 15 of their papers we have counts for
16 papers
Input Matters: Evaluating Input Structure's Impact on LLM Summaries of Sports Play-by-Play
Barkavi Sundararajan, Somayajulu Sripada, Ehud Reiter
A major concern when deploying LLMs in accuracy-critical domains such as sports reporting is that the generated text may not faithfully reflect the input data. We quantify how inpu…
When LLMs Can't Help: Real-World Evaluation of LLMs in Nutrition
Karen Jia-Hui Li, Simone Balloccu, Ondrej Dusek +1
The increasing trust in large language models (LLMs), especially in the form of chatbots, is often undermined by the lack of their extrinsic evaluation. This holds particularly tru…
SPHERE: An Evaluation Card for Human-AI Systems
Qianou Ma, Dora Zhao, Xinran Zhao +6
In the era of Large Language Models (LLMs), establishing effective evaluation methods and standards for diverse human-AI interaction systems is increasingly challenging. To encoura…
PhilHumans: Benchmarking Machine Learning for Personal Health
Vadim Liventsev, Vivek Kumar, Allmin Pradhap Singh Susaiyah +14
The use of machine learning in Healthcare has the potential to improve patient outcomes as well as broaden the reach and affordability of Healthcare. The history of other applicati…
Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo
Barkavi Sundararajan, Somayajulu Sripada, Ehud Reiter
Neural Table-to-Text models tend to hallucinate, producing texts that contain factual errors. We investigate whether such errors in the output can be traced back to problems with t…
Textual Summarisation of Large Sets: Towards a General Approach
Kittipitch Kuptavanich, Ehud Reiter, Kees Van Deemter +1
We are developing techniques to generate summary descriptions of sets of objects. In this paper, we present and evaluate a rule-based NLG technique for summarising sets of bibliogr…
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
Anya Belz, Craig Thomson, Ehud Reiter +39
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/le…
Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation
Aleksandar Savkov, Francesco Moramarco, Alex Papadopoulos Korfiatis +3
Evaluating automatically generated text is generally hard due to the inherently subjective nature of many aspects of the output quality. This difficulty is compounded in automatic…
Comparing informativeness of an NLG chatbot vs graphical app in diet-information domain
Simone Balloccu, Ehud Reiter
Visual representation of data like charts and tables can be challenging to understand for readers. Previous work showed that combining visualisations with text can improve the comm…
User-Driven Research of Medical Note Generation Software
Tom Knoll, Francesco Moramarco, Alex Papadopoulos Korfiatis +7
A growing body of work uses Natural Language Processing (NLP) methods to automatically generate medical notes from audio recordings of doctor-patient consultations. However, there…
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera +5
In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated c…
Generation Challenges: Results of the Accuracy Evaluation Shared Task
Craig Thomson, Ehud Reiter
The Shared Task on Evaluating Accuracy focused on techniques (both manual and automatic) for evaluating the factual accuracy of texts produced by neural NLG systems, in a sports-re…
Towards objectively evaluating the quality of generated medical summaries
Francesco Moramarco, Damir Juric, Aleksandar Savkov +1
We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We…
A preliminary study on evaluating Consultation Notes with Post-Editing
Francesco Moramarco, Alex Papadopoulos Korfiatis, Aleksandar Savkov +1
Automatic summarisation has the potential to aid physicians in streamlining clerical tasks such as note taking. But it is notoriously difficult to evaluate these systems and demons…
Natural Language Generation Challenges for Explainable AI
Ehud Reiter
Good quality explanations of artificial intelligence (XAI) reasoning must be written (and evaluated) for an explanatory purpose, targeted towards their readers, have a good narrati…
Acquiring Correct Knowledge for Natural Language Generation
E. Reiter, R. Robertson, S. G. Sripada
Natural language generation (NLG) systems are computer software systems that produce texts in English and other human languages, often from non-linguistic input data. NLG systems,…