A Theory-Guided LLM Pedagogical Agent Supporting Open-Ended STEM+C Learning

Abstract

Open-ended learning environments (OELEs) can support STEM+C (science, technology, engineering, mathematics, and computing) inquiry through activities such as computational modeling, which requires students to translate disciplinary knowledge across domains and coordinate scientific concepts with computational representations. However, this is often challenging for students, many of whom require support during problem solving. In this exploratory study, we present CoSMoAgent (Computational Science Modeling Agent), a GPT-5-based multi-agent knowledgeable peer for high-school kinematics modeling grounded in the Evidence-Decision-Feedback (EDF) framework. CoSMoAgent goes beyond dialogue: it infers students' strategies, learner states, and knowledge gaps from environment logs and evolving model states. These inferences guide dialogue policies that prioritize probing and hints over direct answers. We deployed CoSMoAgent with 11 dyads across three modeling tasks (33 dyad-sessions), using a mixed-methods design to examine learning outcomes, processes, and adaptive scaffolding. As task mastery increased, students verbalized their understanding more successfully and sought help less often, and the agent's policies shifted from probing toward suggesting actions and extending reasoning. Two contrasting case-study dyads followed markedly different problem-solving paths, yet achieved similar learning gains despite substantial differences in task mastery and CoSMoAgent interaction. Students valued the agent's questions but were often frustrated by its withholding of answers. The study contributes a design pattern and classroom evidence for LLM scaffolding grounded in environmental activity rather than chat alone within an open-ended modeling environment.

arXiv commentSubmitted to Educational Technology Research and Development (ETR&D). Currently under review