20 citations · 20 across the 4 of their papers we have counts for
4 papers · 1 filter
PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks
Claas Beger, Ryan Yi, Melanie Mitchell
The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI models. Yet standard ARC evaluati…
Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks
Adrien Deliège, Claas Beger, Marc Van Droogenbroeck +1
The Abstraction and Reasoning Corpus and related benchmarks evaluate whether AI models can solve novel reasoning tasks, but often leave unclear whether success reflects inference o…
Distinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning
Claas Beger, Ryan Yi, Shuhao Fu +5
AI reasoning models have exceeded human performance on the ARC-AGI-1 benchmark, but does that mean state-of-the-art models have the underlying competence (humanlike abstract reason…
Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks
Melanie Mitchell, Alessandro B. Palmarini, Arseny Moskvichev
We explore the abstract reasoning abilities of text-only and multimodal versions of GPT-4, using the ConceptARC benchmark [10], which is designed to evaluate robust understanding a…