5 papers
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
Bang Nguyen, Dominik Soós, Qian Ma +8
The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers. Existing benchmarks focus primarily on the computational aspect of thi…
KnowledgeGain: Evaluating and Optimizing Science News Generation for Reader Learning
Dominik Soós, Meng Jiang, Jian Wu
Science news is an important medium to communicate discoveries between the research communities and the public. Yet, most metrics for generated or summarized text evaluate semantic…
Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles
Sai Koneru, Jian Wu, Sarah Rajtmajer
Extracting hypotheses and their supporting statistical evidence from full-text scientific articles is central to the synthesis of empirical findings, but remains difficult due to d…
Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Models
Sai Koneru, Elphin Joe, Christine Kirchhoff +2
In contested domains, instruction-tuned language models must balance user-alignment pressures against faithfulness to the in-context evidence. To evaluate this tension, we introduc…
What Are Research Hypotheses?
Jian Wu, Sarah Rajtmajer
Over the past decades, alongside advancements in natural language processing, significant attention has been paid to training models to automatically extract, understand, test, and…