Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers

Abstract

Motivation: Students learning to program often reach states where they are stuck and can make no forward progress. An automatically generated next-step hint can help them make forward progress and support their learning. It is important to know what makes a good hint or a bad hint, and how to generate good hints automatically in novice programming tools, for example using Large Language Models (LLMs). Method and participants: We recruited 44 Java educators from around the world to participate in an online study. We used a set of real student code states as hint-generation scenarios. Participants used a technique known as comparative judgement to rank a set of candidate next-step Java hints, which were generated by Large Language Models (LLMs) and by five human experienced educators. Participants ranked the hints without being told how they were generated. Findings: We found that LLMs had considerable variation in generating high quality next-step hints for programming novices, with GPT-4 outperforming other models tested. When used with a well-designed prompt, GPT-4 outperformed human experts in generating pedagogically valuable hints. A multi-stage prompt was the most effective LLM prompt. We found that the two most important factors of a good hint were length (80--160 words being best), and reading level (US grade 9 or below being best). Offering alternative approaches to solving the problem was considered bad, and we found no effect of sentiment. Conclusions: Automatic generation of these hints is immediately viable, given that LLMs outperformed humans -- even when the students' task is unknown. The fact that only the best prompts achieve this outcome suggests that students on their own are unlikely to be able to produce the same benefit. The prompting task, therefore, should be embedded in an expert-designed tool.

36 pages, 14 figures, 7 tables

References in corpus (13)

• ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R
• Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
• Using Large Language Models to Enhance Programming Error Messages
• "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
• Comparing Code Explanations Created by Students and Large Language Models
• Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
• Desirable Characteristics for AI Teaching Assistants in Programming Education
• Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
• How Beginning Programmers and Code LLMs (Mis)read Each Other
• A Survey of Automated Programming Hint Generation -- The HINTS Framework
• Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices
• Navigating Compiler Errors with AI Assistance -- A Study of GPT Hints in an Introductory Programming Course
• One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks