18 citations · 31 across the 14 of their papers we have counts for
5 papers · 1 filter
SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models
Berke Arda, Peng Cui, Qiaoyuan Zheng +3
Current evaluation of LLMs on benchmarks typically aggregates performance over individual examples into a single scalar score. This approach is overly coarse and ignores the divers…
CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets
Berke Arda, Ahmetcan Yavuz, Paul Gerry +7
Croissant has emerged as a standard for machine-readable dataset metadata, yet populating its fields remains labor-intensive and requires careful reading of accompanying dataset do…
Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Jan Batzner, Sree Harsha Nelaturu, Damian Stachura +45
AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First…
Benchmarking and Enhancing Text-to-Image Models for Generating Visual Representations in Early Arithmetic Education
Junling Wang, Boqi Chen, Heejin Do +3
AI systems are increasingly used to support educational content creation, yet it remains unclear whether they can generate outputs that faithfully represent the pedagogical concept…
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Mubashara Akhtar, Anka Reuel, Prajna Soni +34
Artificial intelligence benchmarks are an important mechanism to measure model progress and guide deployment decisions. However, benchmarks quickly "saturate", making it difficult…