85 citations · 533 across the 53 of their papers we have counts for
12 papers · 1 filter
Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-End
Yanran Chen, Steffen Eger
We consider the end-to-end abstract-to-title generation problem, exploring seven recent transformer based models (including ChatGPT) fine-tuned on more than 30k abstract-title pair…
ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models
Jonas Belouadi, Steffen Eger
State-of-the-art poetry generation systems are often complex. They either consist of task-specific model pipelines, incorporate prior knowledge in the form of manually created cons…
BMX: Boosting Natural Language Generation Metrics with Explainability
Christoph Leiter, Hoa Nguyen, Steffen Eger
State-of-the-art natural language generation evaluation metrics are based on black-box language models. Hence, recent works consider their explainability with the goals of better u…
Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?
Doan Nam Long Vu, Nafise Sadat Moosavi, Steffen Eger
The evaluation of recent embedding-based evaluation metrics for text generation is primarily based on measuring their correlation with human evaluations on standard benchmarks. How…
EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics
Daniil Larionov, Jens Grünwald, Christoph Leiter +1
Efficiency is a key property to foster inclusiveness and reduce environmental costs, especially in an era of LLMs. In this work, we provide a comprehensive evaluation of efficiency…
MENLI: Robust Evaluation Metrics from Natural Language Inference
Yanran Chen, Steffen Eger
Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating to information cor…
Reproducibility Issues for BERT-based Evaluation Metrics
Yanran Chen, Jonas Belouadi, Steffen Eger
Reproducibility is of utmost concern in machine learning and natural language processing (NLP). In the field of natural language generation (especially machine translation), the se…
Towards Explainable Evaluation Metrics for Natural Language Generation
Christoph Leiter, Piyawat Lertvittayakumjorn, Marina Fomicheva +3
Unlike classical lexical overlap metrics such as BLEU, most current evaluation metrics (such as BERTScore or MoverScore) are based on black-box language models such as BERT or XLM-…
Did AI get more negative recently?
Dominik Beese, Begüm Altunbaş, Görkem Güzeler +1
In this paper, we classify scientific articles in the domain of natural language processing (NLP) and machine learning (ML), as core subfields of artificial intelligence (AI), into…
USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation
Jonas Belouadi, Steffen Eger
The vast majority of evaluation metrics for machine translation are supervised, i.e., (i) are trained on human scores, (ii) assume the existence of reference translations, or (iii)…
Constrained Density Matching and Modeling for Cross-lingual Alignment of Contextualized Representations
Wei Zhao, Steffen Eger
Multilingual representations pre-trained with monolingual data exhibit considerably unequal task performances across languages. Previous studies address this challenge with resourc…
DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence
Wei Zhao, Michael Strube, Steffen Eger
Recently, there has been a growing interest in designing text generation systems from a discourse coherence perspective, e.g., modeling the interdependence between sentences. Still…