Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat

Abstract

Large Language Models (LLMs) have the potential to fundamentally change the way people engage in computer programming. Agent-based modeling (ABM) has become ubiquitous in natural and social sciences and education, yet no prior studies have explored the potential of LLMs to assist it. We designed NetLogo Chat to support the learning and practice of NetLogo, a programming language for ABM. To understand how users perceive, use, and need LLM-based interfaces, we interviewed 30 participants from global academia, industry, and graduate schools. Experts reported more perceived benefits than novices and were more inclined to adopt LLMs in their workflow. We found significant differences between experts and novices in their perceptions, behaviors, and needs for human-AI collaboration. We surfaced a knowledge gap between experts and novices as a possible reason for the benefit gap. We identified guidance, personalization, and integration as major needs for LLM-based interfaces to support the programming of ABM.

Conditionally accepted (with minor revisions) by Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24)

References in corpus (11)

• What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
• The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
• Generative AI Assistants in Software Development Education: A vision for integrating Generative AI into educational practice, not instinctively defending against it
• Beyond Generating Code: Evaluating GPT on a Data Visualization Course
• MyCrunchGPT: A chatGPT assisted framework for scientific machine learning
• Using an LLM to Help With Code Understanding
• MultiCoder: Multi-Programming-Lingual Pre-Training for Low-Resource Code Completion

Cited by in corpus (5)

• Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward
• What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
• MetaAgents: Large Language Model Based Agents for Decision-Making on Teaming
• Building babyGPTs: Youth Engaging in Data Practices and Ethical Considerations through the Construction of Generative Language Models
• Using (Not-so) Large Language Models to Generate Simulation Models in a Formal DSL: A Study on Reaction Networks