85 citations · 533 across the 53 of their papers we have counts for
61 papers
NLLG Quarterly arXiv Report 06/23: What are the most influential current AI Papers?
Steffen Eger, Christoph Leiter, Jonas Belouadi +5
The rapid growth of information in the field of Generative Artificial Intelligence (AI), particularly in the subfields of Natural Language Processing (NLP) and Machine Learning (ML…
Cross-Genre Argument Mining: Can Language Models Automatically Fill in Missing Discourse Markers?
Gil Rocha, Henrique Lopes Cardoso, Jonas Belouadi +1
Available corpora for Argument Mining differ along several axes, and one of the key differences is the presence (or absence) of discourse markers to signal argumentative content. E…
Towards Explainable Evaluation Metrics for Machine Translation
Christoph Leiter, Piyawat Lertvittayakumjorn, Marina Fomicheva +3
Unlike classical lexical overlap metrics such as BLEU, most current evaluation metrics for machine translation (for example, COMET or BERTScore) are based on black-box large langua…
Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluation
Ran Zhang, Jihed Ouni, Steffen Eger
While summarization has been extensively researched in natural language processing (NLP), cross-lingual cross-temporal summarization (CLCTS) is a largely unexplored area that has t…
Trade-Offs Between Fairness and Privacy in Language Modeling
Cleo Matzken, Steffen Eger, Ivan Habernal
Protecting privacy in contemporary NLP models is gaining in importance. So does the need to mitigate social biases of such models. But can we have both at the same time? Existing r…
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
Anya Belz, Craig Thomson, Ehud Reiter +39
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/le…
ChatGPT: A Meta-Analysis after 2.5 Months
Christoph Leiter, Ran Zhang, Yanran Chen +4
ChatGPT, a chatbot developed by OpenAI, has gained widespread popularity and media attention since its release in November 2022. However, little hard evidence is available regardin…
Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End
Yanran Chen, Steffen Eger
We consider the end-to-end abstract-to-title generation problem, exploring seven recent transformer based models (including ChatGPT) fine-tuned on more than 30k abstract-title pair…
ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models
Jonas Belouadi, Steffen Eger
State-of-the-art poetry generation systems are often complex. They either consist of task-specific model pipelines, incorporate prior knowledge in the form of manually created cons…
BMX: Boosting Natural Language Generation Metrics with Explainability
Christoph Leiter, Hoa Nguyen, Steffen Eger
State-of-the-art natural language generation evaluation metrics are based on black-box language models. Hence, recent works consider their explainability with the goals of better u…
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Doan Nam Long Vu, Nafise Sadat Moosavi, Steffen Eger
The evaluation of recent embedding-based evaluation metrics for text generation is primarily based on measuring their correlation with human evaluations on standard benchmarks. How…
EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics
Daniil Larionov, Jens Grünwald, Christoph Leiter +1
Efficiency is a key property to foster inclusiveness and reduce environmental costs, especially in an era of LLMs. In this work, we provide a comprehensive evaluation of efficiency…
MENLI: Robust Evaluation Metrics from Natural Language Inference
Yanran Chen, Steffen Eger
Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information cor…
Reproducibility Issues for BERT-based Evaluation Metrics
Yanran Chen, Jonas Belouadi, Steffen Eger
Reproducibility is of utmost concern in machine learning and natural language processing (NLP). In the field of natural language generation (especially machine translation), the se…
Towards Explainable Evaluation Metrics for Natural Language Generation
Christoph Leiter, Piyawat Lertvittayakumjorn, Marina Fomicheva +3
Unlike classical lexical overlap metrics such as BLEU, most current evaluation metrics (such as BERTScore or MoverScore) are based on black-box language models such as BERT or XLM-…
Did AI get more negative recently?
Dominik Beese, Begüm Altunbaş, Görkem Güzeler +1
In this paper, we classify scientific articles in the domain of natural language processing (NLP) and machine learning (ML), as core subfields of artificial intelligence (AI), into…
USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation
Jonas Belouadi, Steffen Eger
The vast majority of evaluation metrics for machine translation are supervised, i.e., (i) are trained on human scores, (ii) assume the existence of reference translations, or (iii)…
Constrained Density Matching and Modeling for Cross-lingual Alignment of Contextualized Representations
Wei Zhao, Steffen Eger
Multilingual representations pre-trained with monolingual data exhibit considerably unequal task performances across languages. Previous studies address this challenge with resourc…
DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence
Wei Zhao, Michael Strube, Steffen Eger
Recently, there has been a growing interest in designing text generation systems from a discourse coherence perspective, e.g., modeling the interdependence between sentences. Still…
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Marvin Kaster, Wei Zhao, Steffen Eger
Evaluation metrics are a key ingredient for progress of text generation systems. In recent years, several BERT-based evaluation metrics have been proposed (including BERTScore, Mov…
Better than Average: Paired Evaluation of NLP Systems
Maxime Peyrard, Wei Zhao, Steffen Eger +1
Evaluation in NLP is usually done by comparing the scores of competing systems independently averaged over a common set of test instances. In this work, we question the use of aver…
The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results
Marina Fomicheva, Piyawat Lertvittayakumjorn, Wei Zhao +2
In this paper, we introduce the Eval4NLP-2021shared task on explainable quality estimation. Given a source-translation pair, this shared task requires not only to provide a sentenc…
Diachronic Analysis of German Parliamentary Proceedings: Ideological Shifts through the Lens of Political Biases
Tobias Walter, Celina Kirschner, Steffen Eger +3
We analyze bias in historical corpora as encoded in diachronic distributional semantic models by focusing on two specific forms of bias, namely a political (i.e., anti-communism) a…
Changes in European Solidarity Before and During COVID-19: Evidence from a Large Crowd- and Expert-Annotated Twitter Dataset
Alexandra Ils, Dan Liu, Daniela Grunow +1
We introduce the well-established social scientific concept of social solidarity and its contestation, anti-solidarity, as a new problem setting to supervised machine learning in N…