Critically Engaged Pragmatism: Scientific Norm and Social, Pragmatist Epistemology for AI Science Evaluation Tools

Abstract

AI science evaluation tools aim to assess research credibility. As with traditional metrics such as impact factors, their edicts can be decontextualised and repurposed in problematic ways. To address this, I propose Critically-Engaged Pragmatism as a scientific norm enjoining scientific communities to scrutinise the purposes and purpose-specific reliability of AI science evaluation tools. To foster Critically Engaged Pragmatism, creators of AI science evaluation tools should transparently and fully report design, training, and benchmarking details to facilitate assessments of purpose-specific reliability, liability to different types of error, and bias. What count as best practices for the transparent reporting of AI science evaluation tools should be updated as new forms of error, bias, and gamesmanship are discovered. Under this framework, AI science evaluation tools are not objective arbiters of scientific credibility. Rather, they are the object of critical discursive practices that ultimately ground the credibility of scientific communities.

References in corpus (7)

• Jury Learning: Integrating Dissenting Voices into Machine Learning Models
• Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
• Is preprint the future of science? A thirty year journey of online preprint services
• A Roadmap to Pluralistic Alignment
• CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
• AAAR-1.0: Assessing AI's Potential to Assist Research
• Detecting Reference Errors in Scientific Literature with Large Language Models