"It's not like Jarvis, but it's pretty close!" -- Examining ChatGPT's Usage among Undergraduate Students in Computer Science
arXiv:2311.09651 · doi:10.1145/3636243.3636257
Abstract
Large language models (LLMs) such as ChatGPT and Google Bard have garnered significant attention in the academic community. Previous research has evaluated these LLMs for various applications such as generating programming exercises and solutions. However, these evaluations have predominantly been conducted by instructors and researchers, not considering the actual usage of LLMs by students. This study adopts a student-first approach to comprehensively understand how undergraduate computer science students utilize ChatGPT, a popular LLM, released by OpenAI. We employ a combination of student surveys and interviews to obtain valuable insights into the benefits, challenges, and suggested improvements related to ChatGPT. Our findings suggest that a majority of students (over 57%) have a convincingly positive outlook towards adopting ChatGPT as an aid in coursework-related tasks. However, our research also highlights various challenges that must be resolved for long-term acceptance of ChatGPT amongst students. The findings from this investigation have broader implications and may be applicable to other LLMs and their role in computing education.
Accepted in ACE 2024: https://aceconference2024.github.io/aceconference2024/ We thank Shamik Sinha, Palak Bhardwaj, Rayyan Hussain, Kashvi Panvanda, Sidhartha Garg, Shagun Yadav, Yash Chillar, and Sourav Goyal for their assistance in the development of questionnaires, as well as their involvement in conducting selected interviews
References in corpus (12)
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
- Visual Instruction Tuning
- Automatic Generation of Programming Exercises and Code Explanations using Large Language Models
- On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?
- A Survey on In-context Learning
- Using Large Language Models to Enhance Programming Error Messages
- Comparing Code Explanations Created by Students and Large Language Models
- Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?
- Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?
- Effects of Human vs. Automatic Feedback on Students' Understanding of AI Concepts and Programming Style
- Better Zero-Shot Reasoning with Role-Play Prompting