Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants

Abstract

AI assistants are being increasingly used by students enrolled in higher education institutions. While these tools provide opportunities for improved teaching and education, they also pose significant challenges for assessment and learning outcomes. We conceptualize these challenges through the lens of vulnerability, the potential for university assessments and learning outcomes to be impacted by student use of generative AI. We investigate the potential scale of this vulnerability by measuring the degree to which AI assistants can complete assessment questions in standard university-level STEM courses. Specifically, we compile a novel dataset of textual assessment questions from 50 courses at EPFL and evaluate whether two AI assistants, GPT-3.5 and GPT-4 can adequately answer these questions. We use eight prompting strategies to produce responses and find that GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions. When grouping courses in our dataset by degree program, these systems already pass non-project assessments of large numbers of core courses in various degree programs, posing risks to higher education accreditation that will be amplified as these models improve. Our results call for revising program-level assessment design in higher education in light of advances in generative AI.

20 pages, 8 figures

References in corpus (14)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Language Models are Few-Shot Learners
• Large Language Models are Zero-Shot Reasoners
• Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality
• Exploring the Role of AI Assistants in Computer Science Education: Methods, Implications, and Instructor Perspectives
• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants
• REFINER: Reasoning Feedback on Intermediate Representations
• LLM Evaluators Recognize and Favor Their Own Generations
• Prompts First, Finally

Cited by in corpus (1)

• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants