61 citations · 145 across the 14 of their papers we have counts for
18 papers
Natural Language Generation
Emiel van Miltenburg, Chenghua Lin
This article provides a brief overview of the field of Natural Language Generation. The term Natural Language Generation (NLG), in its broadest definition, refers to the study of s…
Dual use issues in the field of Natural Language Generation
Emiel van Miltenburg
This report documents the results of a recent survey in the SIGGEN community, focusing on Dual Use issues in Natural Language Generation (NLG). SIGGEN is the Special Interest Group…
Image captioning in different languages
Emiel van Miltenburg
This short position paper provides a manually curated list of non-English image captioning datasets (as of May 2024). Through this list, we can observe the dearth of datasets in di…
Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations
Anouck Braggaar, Christine Liebrecht, Emiel van Miltenburg +1
This review gives an extensive overview of evaluation methods for task-oriented dialogue systems, paying special attention to practical applications of dialogue systems, for exampl…
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
Anya Belz, Craig Thomson, Ehud Reiter +39
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/le…
Evaluating NLG systems: A brief introduction
Emiel van Miltenburg
This year the International Conference on Natural Language Generation (INLG) will feature an award for the paper with the best evaluation. The purpose of this award is to provide a…
Implicit causality in GPT-2: a case study
Hien Huynh, Tomas O. Lentz, Emiel van Miltenburg
This case study investigates the extent to which a language model (GPT-2) is able to capture native speakers' intuitions about implicit causality in a sentence completion task. We…
Underreporting of errors in NLG output, and what to do about it
Emiel van Miltenburg, Miruna-Adriana Clinciu, Ondřej Dušek +8
We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator o…
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets
Simon Mille, Kaustubh D. Dhole, Saad Mahamood +5
Machine learning approaches applied to NLP are often evaluated by summarizing their performance in a single number, for example accuracy. Since most test sets are constructed as an…
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal +53
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of auto…
Preregistering NLP Research
Emiel van Miltenburg, Chris van der Lee, Emiel Krahmer
Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This practice is increasing…
On the use of human reference data for evaluating automatic image descriptions
Emiel van Miltenburg
Automatic image description systems are commonly trained and evaluated using crowdsourced, human-generated image descriptions. The best-performing system is then determined using s…
Neural data-to-text generation: A comparison between pipeline and end-to-end architectures
Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg +1
Traditionally, most data-to-text applications have been designed using a modular pipeline architecture, in which non-linguistic input data is converted into natural language throug…
Cross-linguistic differences and similarities in image descriptions
Emiel van Miltenburg, Desmond Elliott, Piek Vossen
Automatic image description systems are commonly trained and evaluated on large image description datasets. Recently, researchers have started to collect such datasets for language…
Room for improvement in automatic image description: an error analysis
Emiel van Miltenburg, Desmond Elliott
In recent years we have seen rapid and significant progress in automatic image description but what are the open problems in this area? Most work has been evaluated using text-base…
Pragmatic factors in image description: the case of negations
Emiel van Miltenburg, Roser Morante, Desmond Elliott
We provide a qualitative analysis of the descriptions containing negations (no, not, n't, nobody, etc) in the Flickr30K corpus, and a categorization of negation uses. Based on this…
Stereotyping and Bias in the Flickr30K Dataset
Emiel van Miltenburg
An untested assumption behind the crowdsourced descriptions of the images in the Flickr30K dataset (Young et al., 2014) is that they "focus only on the information that can be obta…
Detecting and ordering adjectival scalemates
Emiel van Miltenburg
This paper presents a pattern-based method that can be used to infer adjectival scales, such as <lukewarm, warm, hot>, from a corpus. Specifically, the proposed method uses lexical…