8 citations · 23 across the 10 of their papers we have counts for
11 papers
LLMs as Span Annotators: A Comparative Study of LLMs and Humans
Zdeněk Kasner, Vilém Zouhar, Patrícia Schmidtová +7
Span annotation - annotating specific text features at the span level - can be used to evaluate texts where single-score metrics fail to provide actionable feedback. Until recently…
Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices
Patrícia Schmidtová, Saad Mahamood, Simone Balloccu +6
Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners…
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
Anya Belz, Craig Thomson, Ehud Reiter +39
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/le…
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran +74
Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using…
You Are What You Write: Preserving Privacy in the Era of Large Language Models
Richard Plant, Valerio Giuffrida, Dimitra Gkatzia
Large scale adoption of large language models has introduced a new era of convenient knowledge transfer for a slew of natural language processing tasks. However, these models also…
Task2Dial: A Novel Task and Dataset for Commonsense enhanced Task-based Dialogue Grounded in Documents
Carl Strathearn, Dimitra Gkatzia
This paper proposes a novel task on commonsense-enhanced task-based dialogue grounded in documents and describes the Task2Dial dataset, a novel dataset of document-grounded task-ba…
CAPE: Context-Aware Private Embeddings for Private Language Learning
Richard Plant, Dimitra Gkatzia, Valerio Giuffrida
Deep learning-based language models have achieved state-of-the-art results in a number of applications including sentiment analysis, topic labelling, intent classification and othe…
Underreporting of errors in NLG output, and what to do about it
Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondřej Dušek +8
We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator o…
Content Selection in Data-to-Text Systems: A Survey
Dimitra Gkatzia
Data-to-text systems are powerful in generating reports from data automatically and thus they simplify the presentation of complex data. Rather than presenting data using visualisa…
Natural Language Generation enhances human decision-making with uncertain information
Dimitra Gkatzia, Oliver Lemon, Verena Rieser
Decision-making is often dependent on uncertain data, e.g. data associated with confidence scores or probabilities. We present a comparison of different information presentations f…
An Ensemble method for Content Selection for Data-to-text Systems
Dimitra Gkatzia, Helen Hastie
We present a novel approach for automatic report generation from time-series data, in the context of student feedback generation. Our proposed methodology treats content selection…