7 citations · 12 across the 10 of their papers we have counts for
11 papers
CruxBench: A Benchmark of Information Discovery
Hui Dai, Lina Piao, Nick Merrill +3
Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many complex real-world tasks is identi…
Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments
Christopher W. Karvetski, Sheldon S. Huang, Simas Kučinskas +4
Decision-makers routinely rely on expert judgments accompanied by written explanations, yet explanation quality is difficult to measure at scale. Forecasting tournaments offer a na…
ForecastBench-Sim: A Simulated-World Forecasting Benchmark
Jaeho Lee, Nick Merrill, Ezra Karger
Forecasting benchmarks for general-purpose AI systems usually inherit the constraints of the real world: outcomes resolve slowly, tail events are rare, and counterfactual questions…
Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most
Nick Merrill, Jaeho Lee, Ezra Karger
We document inverse scaling in LLMs on forecasting problems whose underlying time series exhibit superlinear growth and tail risk of regime change, a structure common in finance an…
WOMAC: A Mechanism For Prediction Competitions
Siddarth Srinivasan, Tao Lin, Connacher Murphy +3
Competitions are widely used to identify top performers in judgmental forecasting and machine learning, and the standard competition design ranks competitors based on their cumulat…
When Large Language Models are More PersuasiveThan Incentivized Humans, and Why
Jiacheng Liu, Francesco Salvi, Philipp Schoenegger +37
Large Language Models (LLMs) have been shown to be highly persuasive, but when and why they outperform humans is still an open question. We compare the persuasiveness of two LLMs (…
Tell Me Why: Incentivizing Explanations
Siddarth Srinivasan, Ezra Karger, Michiel Bakker +1
Common sense suggests that when individuals explain why they believe something, we can arrive at more accurate conclusions than when they simply state what they believe. Yet, there…
Identifying good forecasters via adaptive cognitive tests
Edgar C. Merkle, Nikolay Petrov, Sophie Ma Zhu +3
Assessing forecasting performance is a time intensive activity, often requiring months or years before we know whether or not the reported forecasts were accurate. Cognitive tests…
ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities
Ezra Karger, Houtan Bastani, Chen Yueh-Han +4
Forecasts of future events are essential inputs into informed decision-making. Machine learning (ML) systems have the potential to deliver forecasts at scale, but there is no frame…
AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy
Philipp Schoenegger, Peter S. Park, Ezra Karger +2
Large language models (LLMs) match and sometimes exceeding human performance in many domains. This study explores the potential of LLMs to augment human judgement in a forecasting…
Self-Resolving Prediction Markets for Unverifiable Outcomes
Siddarth Srinivasan, Ezra Karger, Yiling Chen
Prediction markets elicit and aggregate beliefs by paying agents based on how close their predictions are to a verifiable future outcome. However, outcomes of many important questi…