137 citations · 212 across the 21 of their papers we have counts for
4 papers · 1 filter
Schroedinger's Threshold: When the AUC doesn't predict Accuracy
Juri Opitz
The Area Under Curve measure (AUC) seems apt to evaluate and compare diverse models, possibly without calibration. An important example of AUC application is the evaluation and ben…
Natural Language Processing RELIES on Linguistics
Juri Opitz, Shira Wein, Nathan Schneider
Large Language Models (LLMs) have become capable of generating highly fluent text in certain languages, without modules specially designed to capture grammar or semantic coherence.…
On the Role of Summary Content Units in Text Summarization Evaluation
Marcel Nawrath, Agnieszka Nowak, Tristan Ratz +13
At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into…
A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice
Juri Opitz
Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Frequently, metrics are selected without argument…