33 citations · 54 across the 18 of their papers we have counts for
6 papers · 1 filter
Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluation
Mingqi Gao, Xinyu Hu, Li Lin +1
The correlation between NLG automatic evaluation metrics and human evaluation is often regarded as a critical criterion for assessing the capability of an evaluation metric. Howeve…
Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling
Jie Ruan, Xiao Pu, Mingqi Gao +2
Human evaluation is viewed as a reliable evaluation method for NLG which is expensive and time-consuming. To save labor and costs, researchers usually perform human evaluation on a…
Defining and Detecting Vulnerability in Human Evaluation Guidelines: A Preliminary Study Towards Reliable NLG Evaluation
Jie Ruan, Wenqing Wang, Xiaojun Wan
Human evaluation serves as the gold standard for assessing the quality of Natural Language Generation (NLG) systems. Nevertheless, the evaluation guideline, as a pivotal element en…
Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability
Xinyu Hu, Li Lin, Mingqi Gao +2
The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area. With the recent emergence of powerful large language models (LLMs), some…
Are LLM-based Evaluators Confusing NLG Quality Criteria?
Xinyu Hu, Mingqi Gao, Sen Hu +4
Some prior work has shown that LLMs perform well in NLG evaluation for different tasks. However, we discover that LLMs seem to confuse different evaluation criteria, which reduces…
LLM-based NLG Evaluation: Current Status and Challenges
Mingqi Gao, Xinyu Hu, Jie Ruan +2
Evaluating natural language generation (NLG) is a vital but challenging problem in natural language processing. Traditional evaluation metrics mainly capturing content (e.g. n-gram…