The Death of the Short-Form Physics Essay in the Coming AI Revolution

Abstract

The latest AI language modules can produce original, high quality full short-form ($300$-word) Physics essays within seconds. These technologies such as ChatGPT and davinci-003 are freely available to anyone with an internet connection. In this work, we present evidence of AI generated short-form essays achieving first-class grades on an essay writing assessment from an accredited, current university Physics module. The assessment requires students answer five open-ended questions with a short, $300$-word essay each. Fifty AI answers were generated to create ten submissions that were independently marked by five separate markers. The AI generated submissions achieved an average mark of $71\pm 2\%$, in strong agreement with the current module average of $71\pm 5$. A typical AI submission would therefore most-likely be awarded a First Class, the highest classification available at UK universities. Plagiarism detection software returned a plagiarism score between $2\pm 1$% (Grammarly) and $7\pm 2$% (TurnitIn). We argue that these results indicate that current AI MLPs represent a significant threat to the fidelity of short-form essays as an assessment method in Physics courses.

14 pages, 2 figures

Cited by in corpus (11)

• Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
• Better by you, better than me, chatgpt3 as writing assistance in students essays
• How understanding large language models can inform the use of ChatGPT in physics education
• The Impact of AI in Physics Education: A Comprehensive Review from GCSE to University Levels
• Performance of ChatGPT on the Test of Understanding Graphs in Kinematics
• Can ChatGPT and Bard Generate Aligned Assessment Items? A Reliability Analysis against Human Performance
• Chatbot-supported Thesis Writing: An Autoethnographic Report
• Evaluating vision-capable chatbots in interpreting kinematics graphs: a comparative study of free and subscription-based models
• Performance of ChatGPT on tasks involving physics visual representations: the case of the Brief Electricity and Magnetism Assessment
• Can ChatGPT pass a physics degree? Making a case for reformation of assessment of undergraduate degrees
• Using AI Large Language Models for Grading in Education: A Hands-On Test for Physics