565 citations · 801 across the 8 of their papers we have counts for
4 papers · 1 filter
Capabilities Ain't All You Need: Measuring Propensities in AI
Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi +11
AI evaluation has primarily focused on measuring capabilities, with formal approaches inspired from Item Response Theory (IRT) being increasingly applied. Yet propensities - the te…
From Human-Level AI Tales to AI Leveling Human Scales
Peter Romero, Fernando Martínez-Plumed, Zachary R. Tidler +11
Comparing AI models to "human level" is often misleading when benchmark scores are incommensurate or human baselines are drawn from a narrow population. To address this, we propose…
What should an AI assessor optimise for?
Daniel Romero-Alvarado, Fernando Martínez-Plumed, José Hernández-Orallo
An AI assessor is an external, ideally indepen-dent system that predicts an indicator, e.g., a loss value, of another AI system. Assessors can lever-age information from the test r…
Compute and Energy Consumption Trends in Deep Learning Inference
Radosvet Desislavov, Fernando Martínez-Plumed, José Hernández-Orallo
The progress of some AI paradigms such as deep learning is said to be linked to an exponential growth in the number of parameters. There are many studies corroborating these trends…