5 citations · 13 across the 8 of their papers we have counts for
9 papers
QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculi
Anthony G. Cohn, Robert E. Blackwell
We introduce an extensive qualitative spatial and temporal reasoning (QSTR) benchmark for evaluating large language models (LLMs). We pose questions concerning compositional reason…
DecompSR: A dataset for decomposed analyses of compositional multihop spatial reasoning
Lachlan McPheat, Navdeep Kaur, Robert Blackwell +3
We introduce DecompSR, decomposed spatial reasoning, a large benchmark dataset (over 5m datapoints) and generation framework designed to analyse compositional spatial reasoning abi…
Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited
Anthony G Cohn, Robert E Blackwell
We investigate the abilities of 28 Large language Models (LLMs) to reason about cardinal directions (CDs) using a benchmark generated from a set of templates, extensively testing a…
An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning
Navdeep Kaur, Lachlan McPheat, Alessandra Russo +2
In this paper, we examine the use of Conformal Language Modelling (CLM) alongside Answer Set Programming (ASP) to enhance the performance of standard open-weight LLMs on complex mu…
City Models: Past, Present and Future Prospects
Helge Ritter, Otthein Herzog, Kurt Rothermel +2
We attempt to take a comprehensive look at the challenges of representing the spatio-temporal structures and dynamic processes defining a city's overall characteristics. For the ta…
Can Large Language Models Reason about the Region Connection Calculus?
Anthony G Cohn, Robert E Blackwell
Qualitative Spatial Reasoning is a well explored area of Knowledge Representation and Reasoning and has multiple applications ranging from Geographical Information Systems to Robot…
Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores
Robert E. Blackwell, Jon Barry, Anthony G. Cohn
Large language models (LLMs) are stochastic, and not all models give deterministic answers, even when setting temperature to zero with a fixed random seed. However, few benchmark s…
Evaluating the Ability of Large Language Models to Reason about Cardinal Directions
Anthony G Cohn, Robert E Blackwell
We investigate the abilities of a representative set of Large language Models (LLMs) to reason about cardinal directions (CDs). To do so, we create two datasets: the first, co-crea…
An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8
Anthony G Cohn
Qualitative Spatial Reasoning (QSR) is well explored area of Commonsense Reasoning and has multiple applications ranging from Geographical Information Systems to Robotics and Compu…