137 citations · 212 across the 21 of their papers we have counts for
25 papers
Schroedinger's Threshold: When the AUC doesn't predict Accuracy
Juri Opitz
The Area Under Curve measure (AUC) seems apt to evaluate and compare diverse models, possibly without calibration. An important example of AUC application is the evaluation and ben…
Natural Language Processing RELIES on Linguistics
Juri Opitz, Shira Wein, Nathan Schneider
Large Language Models (LLMs) have become capable of generating highly fluent text in certain languages, without modules specially designed to capture grammar or semantic coherence.…
On the Role of Summary Content Units in Text Summarization Evaluation
Marcel Nawrath, Agnieszka Nowak, Tristan Ratz +13
At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into…
A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice
Juri Opitz
Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Frequently, metrics are selected without argument…
The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
Christoph Leiter, Juri Opitz, Daniel Deutsch +3
With an increasing number of parameters and pre-training data, generative large language models (LLMs) have shown remarkable capabilities to solve tasks with minimal or no task-rel…
Gzip versus bag-of-words for text classification
Juri Opitz
The effectiveness of compression in text classification ('gzip') has recently garnered lots of attention. In this note we show that `bag-of-words' approaches can achieve similar or…