82 citations · 110 across the 25 of their papers we have counts for
25 papers
Validating DBpedia Triple Sets for Natural Language Generation
Mark Andrade, Simon Mille, Anya Belz +1
We present a study of the quality of individual DBpedia triples from the perspective of Natural Language Generation, and propose and evaluate an approach for collecting entity-spec…
AutoForest: Automatically Generating Forest Plots from Biomedical Studies with End-to-End Evidence Extraction and Synthesis
Massimiliano Pronesti, Angelo Miculescu, Mohsin Kapdi +8
Systematic reviews rely on forest plots to synthesise quantitative evidence across biomedical studies, but generating them remains a fragmented and labour-intensive process. Resear…
A Comparative Study of Controlled Text Generation Systems Using Level-Playing-Field Evaluation Principles
Michela Lorandi, Anya Belz
Background: Many different approaches to controlled text generation (CTG) have been proposed over recent years, but it is difficult to get a clear picture of which approach perform…
Output Composability of QLoRA PEFT Modules for Plug-and-Play Attribute-Controlled Text Generation
Michela Lorandi, Anya Belz
Parameter-efficient fine-tuning (PEFT) techniques offer task-specific fine-tuning at a fraction of the cost of full fine-tuning, but require separate fine-tuning for every new task…
Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference
Mohammed Sabry, Anya Belz
We study distillation for large language models under explicit compute constraints, with the goal of producing student models that are not only cheaper to train, but structurally e…
Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning
Massimiliano Pronesti, Anya Belz, Yufang Hou
Recent work on reinforcement learning with verifiable rewards (RLVR) has shown that large language models (LLMs) can be substantially improved using outcome-level verification sign…
The QCET Taxonomy of Standard Quality Criterion Names and Definitions for the Evaluation of NLP Systems
Anya Belz, Simon Mille, Craig Thomson
Prior work has shown that two NLP evaluation experiments that report results for the same quality criterion name (e.g. Fluency) do not necessarily evaluate the same aspect of quali…
Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learning
Mohammed Sabry, Anya Belz
Mechanism-targeted synthetic data is increasingly proposed as a way to steer pretraining toward desirable capabilities, but it remains unclear how such interventions should be eval…
Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies
Massimiliano Pronesti, Joao Bettencourt-Silva, Paul Flanagan +4
Extracting scientific evidence from biomedical studies for clinical research questions (e.g., Does stem cell transplantation improve quality of life in patients with medically refr…
QRA++: Quantified Reproducibility Assessment for Common Types of Results in Natural Language Processing
Anya Belz
Reproduction studies reported in NLP provide individual data points which in combination indicate worryingly low levels of reproducibility in the field. Because each reproduction s…
Enhancing Study-Level Inference from Clinical Trial Papers via Reinforcement Learning-Based Numeric Reasoning
Massimiliano Pronesti, Michela Lorandi, Paul Flanagan +3
Systematic reviews in medicine play a critical role in evidence-based decision-making by aggregating findings from multiple studies. A central bottleneck in automating this process…
HEDS 3.0: The Human Evaluation Data Sheet Version 3.0
Anya Belz, Craig Thomson
This paper presents version 3.0 of the Human Evaluation Datasheet (HEDS). This update is the result of our experience using HEDS in the context of numerous recent human evaluation…
Reproducing the Metric-Based Evaluation of a Set of Controllable Text Generation Techniques
Michela Lorandi, Anya Belz
Rerunning a metric-based evaluation should be more straightforward, and results should be closer, than in a human-based evaluation, especially where code and model checkpoints are…
High-quality Data-to-Text Generation for Severely Under-Resourced Languages with Out-of-the-box Large Language Models
Michela Lorandi, Anya Belz
The performance of NLP methods for severely under-resourced languages cannot currently hope to match the state of the art in NLP methods for well resourced languages. We explore th…
Assessing the Portability of Parameter Matrices Trained by Parameter-Efficient Finetuning Methods
Mohammed Sabry, Anya Belz
As the cost of training ever larger language models has grown, so has the interest in reusing previously learnt knowledge. Transfer learning methods have shown how reusing non-task…
Data-to-text Generation for Severely Under-Resourced Languages with GPT-3.5: A Bit of Help Needed from Google Translate
Michela Lorandi, Anya Belz
LLMs like GPT are great at tasks involving English which dominates in their training data. In this paper, we look at how they cope with tasks involving languages that are severely…
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
Anya Belz, Craig Thomson, Ehud Reiter +39
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/le…
PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniques
Mohammed Sabry, Anya Belz
Recent parameter-efficient finetuning (PEFT) techniques aim to improve over the considerable cost of fully finetuning large pretrained language models (PLM). As different PEFT tech…
Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation
Aleksandar Savkov, Francesco Moramarco, Alex Papadopoulos Korfiatis +3
Evaluating automatically generated text is generally hard due to the inherently subjective nature of many aspects of the output quality. This difficulty is compounded in automatic…
User-Driven Research of Medical Note Generation Software
Tom Knoll, Francesco Moramarco, Alex Papadopoulos Korfiatis +7
A growing body of work uses Natural Language Processing (NLP) methods to automatically generate medical notes from audio recordings of doctor-patient consultations. However, there…
Quantified Reproducibility Assessment of NLP Results
Anya Belz, Maja Popović, Simon Mille
This paper describes and tests a method for carrying out quantified reproducibility assessment (QRA) that is based on concepts and definitions from metrology. QRA produces a single…
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera +5
In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated c…
Quantifying Reproducibility in NLP and ML
Anya Belz
Reproducibility has become an intensely debated topic in NLP and ML over recent years, but no commonly accepted way of assessing reproducibility, let alone quantifying it, has so f…
A Systematic Review of Reproducibility Research in Natural Language Processing
Anya Belz, Shubham Agarwal, Anastasia Shimorina +1
Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibili…