Performance of ChatGPT on the Test of Understanding Graphs in Kinematics

Abstract

The well-known artificial intelligence-based chatbot ChatGPT-4 has become able to process image data as input in October 2023. We investigated its performance on the Test of Understanding Graphs in Kinematics to inform the physics education community of the current potential of using ChatGPT in the education process, particularly on tasks that involve graphical interpretation. We found that ChatGPT, on average, performed similarly to students taking a high-school level physics course, but with important differences in the distribution of the correctness of its responses, as well as in terms of the displayed "reasoning" and "visual" abilities. While ChatGPT was very successful at proposing productive strategies for solving the tasks on the test and expressed correct "reasoning" in most of its responses, it had difficulties correctly "seeing" graphs. We suggest that, based on its performance, caution and a critical approach are needed if one intends to use it in the role of a tutor, a model of a student, or a tool for assisting vision-impaired persons in the context of kinematics graphs.

References in corpus (6)

• The Death of the Short-Form Physics Essay in the Coming AI Revolution
• Could an Artificial-Intelligence agent pass an introductory physics course?
• Physics task development of prospective physics teachers using ChatGPT
• Enhancing STEM Learning with ChatGPT and Bing Chat as Objects to Think With: A Case Study
• How understanding large language models can inform the use of ChatGPT in physics education
• Can an AI-tool grade assignments in an introductory physics course?

Cited by in corpus (13)

• A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course
• ChatGPT as a tool for honing teachers' Socratic dialogue skills
• Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventories
• Evaluating vision-capable chatbots in interpreting kinematics graphs: a comparative study of free and subscription-based models
• Cheat sites and artificial intelligence usage in online introductory physics courses: what is the extent and what effect does it have on assessments?
• Performance of ChatGPT on tasks involving physics visual representations: the case of the Brief Electricity and Magnetism Assessment
• Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment
• Using AI Large Language Models for Grading in Education: A Hands-On Test for Physics
• Creating a customisable Socratic AI physics tutor
• Opportunities and Challenges in Harnessing Digital Technology for Effective Teaching and Learning
• The Boiling-Frog Problem of Physics Education
• Using Large Language Models for Solving Thermodynamic Problems
• When AI Evaluates Its Own Work: Validating Learner-Initiated, AI-Generated Physics Practice Problems