Physics task development of prospective physics teachers using ChatGPT

Abstract

The recent advancement of large language models presents numerous opportunities for teaching and learning. Despite widespread public debate regarding the use of large language models, empirical research on their opportunities and risks in education remains limited. In this work, we demonstrate the qualities and shortcomings of using ChatGPT 3.5 for physics task development by prospective teachers. In a randomized controlled trial, 26 prospective physics teacher students were divided into two groups: the first group used ChatGPT 3.5 to develop text-based physics tasks for four different concepts in the field of kinematics for 10th grade high school students, while the second group used a classical textbook to create tasks for the same concepts and target group. The results indicate no difference in task correctness, but students using the textbook achieved a higher clarity and more frequently embedded their questions in a meaningful context. Both groups adapted the level of task difficulty easily to the target group but struggled strongly with sufficient task specificity, i.e., relevant information to solve the tasks were missing. Students using ChatGPT for problem posing rated high system usability but experienced difficulties with output quality. These results provide insights into the opportunities and pitfalls of using large language models in education.

10 pages, 3 figures, 3 tables

Cited by in corpus (9)

• Exploring Generative AI assisted feedback writing for students' written responses to a physics conceptual question with prompt engineering and few-shot learning
• Performance of ChatGPT on the Test of Understanding Graphs in Kinematics
• Harnessing Large Language Models to Enhance Self-Regulated Learning via Formative Feedback
• Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventories
• Assessing Confidence in AI-Assisted Grading of Physics Exams through Psychometrics: An Exploratory Study
• Ethel: A Virtual Teaching Assistant
• Performance of ChatGPT on tasks involving physics visual representations: the case of the Brief Electricity and Magnetism Assessment
• Opportunities and Challenges in Harnessing Digital Technology for Effective Teaching and Learning
• When AI Evaluates Its Own Work: Validating Learner-Initiated, AI-Generated Physics Practice Problems