565 citations · 908 across the 35 of their papers we have counts for
22 papers · 1 filter
Seven simple steps for log analysis in AI systems
Magda Dubois, Ekin Zorer, Maia Hamin +17
AI systems produce large volumes of logs as they interact with tools and users. Analysing these logs can help understand model capabilities, propensities, and behaviours, or assess…
Learning Alternative Ways of Performing a Task
David Nieves, María José Ramírez-Quintana, Carlos Monserrat +2
A common way of learning to perform a task is to observe how it is carried out by experts. However, it is well known that for most tasks there is no unique way to perform them. Thi…
Evaluating General-Purpose AI with Psychometrics
Xiting Wang, Liming Jiang, Jose Hernandez-Orallo +4
Comprehensive and accurate evaluation of general-purpose AI systems such as large language models allows for effective mitigation of their risks and deepened understanding of their…
Evaluating the Apperception Engine
Richard Evans, Jose Hernandez-Orallo, Johannes Welbl +2
The Apperception Engine is an unsupervised learning system. Given a sequence of sensory inputs, it constructs a symbolic causal theory that both explains the sensory sequence and a…
Making sense of sensory input
Richard Evans, Jose Hernandez-Orallo, Johannes Welbl +2
This paper attempts to answer a central question in unsupervised learning: what does it mean to "make sense" of a sensory sequence? In our formalization, making sense involves cons…
Analysing Results from AI Benchmarks: Key Indicators and How to Obtain Them
Fernando Martínez-Plumed, José Hernández-Orallo
Item response theory (IRT) can be applied to the analysis of the evaluation of results from AI benchmarks. The two-parameter IRT model provides two indicators (difficulty and discr…
General-purpose Declarative Inductive Programming with Domain-Specific Background Knowledge for Data Wrangling Automation
Lidia Contreras-Ochando, César Ferri, José Hernández-Orallo +3
Given one or two examples, humans are good at understanding how to solve a problem independently of its domain, because they are able to detect what the problem is and to choose th…
A multidisciplinary task-based perspective for evaluating the impact of AI autonomy and generality on the future of work
Enrique Fernández-Macías, Emilia Gómez, José Hernández-Orallo +4
This paper presents a multidisciplinary task approach for assessing the impact of artificial intelligence on the future of work. We provide definitions of a task from two main pers…
Assessing the impact of machine intelligence on human behaviour: an interdisciplinary endeavour
Emilia Gómez, Carlos Castillo, Vicky Charisi +23
This document contains the outcome of the first Human behaviour and machine intelligence (HUMAINT) workshop that took place 5-6 March 2018 in Barcelona, Spain. The workshop was org…
Finite Biased Teaching with Infinite Concept Classes
Jose Hernandez-Orallo, Jan Arne Telle
We investigate the teaching of infinite concept classes through the effect of the learning bias (which is used by the learner to prefer some concepts over others and by the teacher…
Universal Psychometrics Tasks: difficulty, composition and decomposition
Jose Hernandez-Orallo
This note revisits the concepts of task and difficulty. The notion of cognitive task and its use for the evaluation of intelligent systems is still replete with issues. The view of…
Forgetting and consolidation for incremental and cumulative knowledge acquisition systems
Fernando Martínez-Plumed, Cèsar Ferri, José Hernández-Orallo +1
The application of cognitive mechanisms to support knowledge acquisition is, from our point of view, crucial for making the resulting models coherent, efficient, credible, easy to…
A note about the generalisation of the C-tests
Jose Hernandez-Orallo
In this exploratory note we ask the question of what a measure of performance for all tasks is like if we use a weighting of tasks based on a difficulty function. This difficulty f…
AI Evaluation: past, present and future
Jose Hernandez-Orallo
Artificial intelligence develops techniques and systems whose performance must be evaluated on a regular basis in order to certify and foster progress in the discipline. We will de…
On the universality of cognitive tests
David L. Dowe, Jose Hernandez-Orallo
The analysis of the adaptive behaviour of many different kinds of systems such as humans, animals and machines, requires more general ways of assessing their cognitive abilities. T…
A short note on estimating intelligence from user profiles in the context of universal psychometrics: prospects and caveats
Jose Hernandez-Orallo
There has been an increasing interest in inferring some personality traits from users and players in social networks and games, respectively. This goes beyond classical sentiment a…
Complexity distribution of agent policies
Jose Hernandez-Orallo
We analyse the complexity of environments according to the policies that need to be used to achieve high performance. The performance results for a population of policies leads to…
On the influence of intelligence in (social) intelligence testing environments
Javier Insa-Cabrera, Jose-Luis Benacloch-Ayuso, Jose Hernandez-Orallo
This paper analyses the influence of including agents of different degrees of intelligence in a multiagent system. The goal is to better understand how we can develop intelligence…
Threshold Choice Methods: the Missing Link
José Hernández-Orallo, Peter Flach, Cèsar Ferri
Many performance metrics have been introduced for the evaluation of classification performance, with different origins and niches of application: accuracy, macro-accuracy, area und…
Analysis of first prototype universal intelligence tests: evaluating and comparing AI algorithms and humans
Javier Insa-Cabrera, Jose Hernandez-Orallo
Today, available methods that assess AI systems are focused on using empirical techniques to measure the performance of algorithms in some specific tasks (e.g., playing chess, solv…
Technical Note: Towards ROC Curves in Cost Space
José Hernández-Orallo, Peter Flach, Cèsar Ferri
ROC curves and cost curves are two popular ways of visualising classifier performance, finding appropriate thresholds according to the operating condition, and deriving useful aggr…
An architecture for the evaluation of intelligent systems
Javier Insa-Cabrera, Jose Hernandez-Orallo
One of the main research areas in Artificial Intelligence is the coding of agents (programs) which are able to learn by themselves in any situation. This means that agents must be…