6 papers · 1 filter
SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
Lukas Haas, Gal Yona, Giovanni D'Antonio +2
We introduce SimpleQA Verified, a 1,000-prompt benchmark for evaluating Large Language Model (LLM) short-form factuality based on OpenAI's SimpleQA. It addresses critical limitatio…
DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents
Nikita Gupta, Riju Chatterjee, Lukas Haas +9
We introduce DeepSearchQA, a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks across 17 different fields. Unlike traditional benchmarks…
The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality
Aileen Cheng, Alon Jacovi, Amir Globerson +62
We introduce The FACTS Leaderboard, an online leaderboard suite and associated set of benchmarks that comprehensively evaluates the ability of language models to generate factually…
The Bias is in the Details: An Assessment of Cognitive Bias in LLMs
R. Alexander Knipper, Charles S. Knipper, Kaiqi Zhang +3
As Large Language Models (LLMs) are increasingly embedded in real-world decision-making processes, it becomes crucial to examine the extent to which they exhibit cognitive biases.…
The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
Alon Jacovi, Andrew Wang, Chris Alberti +23
We introduce FACTS Grounding, an online leaderboard and associated benchmark that evaluates language models' ability to generate text that is factually accurate with respect to giv…
DOLOMITES: Domain-Specific Long-Form Methodical Tasks
Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev +7
Experts in various fields routinely perform methodical writing tasks to plan, organize, and report their work. From a clinician writing a differential diagnosis for a patient, to a…