Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences

Abstract

Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. Once the evidence is prepared and the calculation is written as code, this choice reduces to one semantic judgment among listed candidates. We evaluate Jev on this step, using a harness that follows its documented guidance. The study compares twelve model configurations on twenty Choices in ten scientific cases built from published sources, each repeated five times. We measure semantic selections, downstream outputs and final claim labels separately. Jev matched five other configurations at complete semantic correctness and achieved the lowest observed median latency and known mean cost among successful responses. Across three comparison models, seven wrong selections on one question about whether samples shared a culture changed downstream counts while preserving the correct final label. These results support using Jev to select relations when the evidence and candidates are prepared in advance, and show that evaluating it requires checking the relations and quantities that a workflow will reuse.

arXiv comment9 pages, 1 figure, 6 tables. Code available at https://github.com/bydeng01/scientific-decision-eval