10 papers
On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists
Seungone Kim, Dongkeun Yoon, Kiril Gashteovski +55
With the advancement of AI capabilities, AI reviewers are beginning to be deployed in scientific peer review, yet their capability and credibility remain in question: many scientis…
Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs
Guijin Son, Seungone Kim, Catherine Arnett +73
Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challenging target for measuring LLM…
DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
Rulin Shao, Akari Asai, Shannon Zejiang Shen +18
Deep research agents perform multi-step research to produce long-form, well-attributed answers. However, most open deep research agents are trained on easily verifiable short-form…
AgentIR: Reasoning-Aware Retrieval for Deep Research Agents
Zijian Chen, Xueguang Ma, Shengyao Zhuang +3
Deep Research agents are rapidly emerging as primary consumers of modern retrieval systems. Unlike human users who issue and refine queries without documenting their intermediate t…
Out of Style: RAG's Fragility to Linguistic Variation
Tianyu Cao, Neel Bhandari, Akhila Yerukola +2
Despite the impressive performance of Retrieval-augmented Generation (RAG) systems across various NLP benchmarks, their robustness in handling real-world user-LLM interaction queri…
Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents
Prahaladh Chandrahasan, Jiahe Jin, Zhihan Zhang +9
Effectively evaluating deep research agents that autonomously search the web, analyze information, and generate reports remains a major challenge, particularly when it comes to ass…