8 papers
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
Bang Nguyen, Dominik Soós, Qian Ma +8
The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers. Existing benchmarks focus primarily on the computational aspect of thi…
The Failed Migration of Academic Twitter: A Case Study of Precocious Adopters
Xinyu Wang, Sai Koneru, Sarah Rajtmajer
Following changes in Twitter's ownership in 2022 and subsequent changes to content moderation policies, many in academia looked to move their discourse elsewhere and migration to M…
Human-AI Collaboration for Estimating Scientific Replicability
Tatiana Chakravorti, Robert Fraleigh, Timothy Fritton +7
Determining whether published scientific findings can successfully be replicated is a long-standing challenge in the empirical sciences. Existing approaches for replicability asses…
Many Ways to Be Fake: Benchmarking Fake News Detection Under Strategy-Driven AI Generation
Xinyu Wang, Sai Koneru, Wenbo Zhang +3
Recent advances in large language models (LLMs) have enabled the large-scale generation of highly fluent and deceptive news-like content. While prior work has often treated fake ne…
Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles
Sai Koneru, Jian Wu, Sarah Rajtmajer
Extracting hypotheses and their supporting statistical evidence from full-text scientific articles is central to the synthesis of empirical findings, but remains difficult due to d…
Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Models
Sai Koneru, Elphin Joe, Christine Kirchhoff +2
In contested domains, instruction-tuned language models must balance user-alignment pressures against faithfulness to the in-context evidence. To evaluate this tension, we introduc…