How Scientists Use Large Language Models to Program

Abstract

Scientists across disciplines write code for critical activities like data collection and generation, statistical modeling, and visualization. As large language models that can generate code have become widely available, scientists may increasingly use these models during research software development. We investigate the characteristics of scientists who are early-adopters of code generating models and conduct interviews with scientists at a public, research-focused university. Through interviews and reviews of user interaction logs, we see that scientists often use code generating models as an information retrieval tool for navigating unfamiliar programming languages and libraries. We present findings about their verification strategies and discuss potential vulnerabilities that may emerge from code generation practices unknowingly influencing the parameters of scientific analyses.

arXiv commentAccepted to CHI 2025

References in corpus (9)

• Best Practices for Scientific Computing
• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• The Metacognitive Demands and Opportunities of Generative AI
• "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
• Perfection Not Required? Human-AI Partnerships in Code Translation
• Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions
• Better Together? An Evaluation of AI-Supported Code Translation
• How Beginning Programmers and Code LLMs (Mis)read Each Other
• Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course