5 papers
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
Bang Nguyen, Dominik Soós, Qian Ma +8
The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers. Existing benchmarks focus primarily on the computational aspect of thi…
CC30k: A Citation Contexts Dataset for Reproducibility-Oriented Sentiment Analysis
Rochana R. Obadage, Sarah M. Rajtmajer, Jian Wu
Sentiments about the reproducibility of cited papers in downstream literature offer community perspectives and have shown as a promising signal of the actual reproducibility of pub…
Toward Robust URL Extraction for Open Science: A Study of arXiv File Formats and Temporal Trends
Rochana R. Obadage, Lamia Salsabil, Sawood Alam +4
In this work, we study how URL extraction results depend on input format. We compiled a pilot dataset by extracting URLs from 10 arXiv papers and used the same heuristic method to…
[Re] Network Deconvolution
Rochana R. Obadage, Kumushini Thennakoon, Sarah M. Rajtmajer +1
Our work aims to reproduce the set of findings published in "Network Deconvolution" by Ye et al. (2020)[1]. That paper proposes an optimization technique for model training in conv…
Can citations tell us about a paper's reproducibility? A case study of machine learning papers
Rochana R. Obadage, Sarah M. Rajtmajer, Jian Wu
The iterative character of work in machine learning (ML) and artificial intelligence (AI) and reliance on comparisons against benchmark datasets emphasize the importance of reprodu…