Can We Count on LLMs? The Fixed-Effect Fallacy and Claims of GPT-4 Capabilities

Thomas Ball, Shuo Chen, Cormac Herley

In this paper we explore evaluation of LLM capabilities. We present measurements of GPT-4 performance on several deterministic tasks; each task involves a basic calculation and tak…