Same Prompt, Different Outcomes: Evaluating the Reproducibility of Data Analysis by LLMs

Jiaxin Cui, Rohan Alexander

We systematically evaluate the reproducibility of data analysis conducted by Large Language Models (LLMs). We evaluate two prompting strategies, six models, and four temperature se…