The Metacognitive Demands and Opportunities of Generative AI

Abstract

Generative AI (GenAI) systems offer unprecedented opportunities for transforming professional and personal work, yet present challenges around prompting, evaluating and relying on outputs, and optimizing workflows. We argue that metacognition$\unicode{x2013}$the psychological ability to monitor and control one's thoughts and behavior$\unicode{x2013}$offers a valuable lens to understand and design for these usability challenges. Drawing on research in psychology and cognitive science, and recent GenAI user studies, we illustrate how GenAI systems impose metacognitive demands on users, requiring a high degree of metacognitive monitoring and control. We propose these demands could be addressed by integrating metacognitive support strategies into GenAI systems, and by designing GenAI systems to reduce their metacognitive demand by targeting explainability and customizability. Metacognition offers a coherent framework for understanding the usability challenges posed by GenAI, and provides novel research and design directions to advance human-AI interaction.

References in corpus (32)

• Survey of Hallucination in Natural Language Generation
• On the Opportunities and Risks of Foundation Models
• What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research
• Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
• The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development
• An Empirical Study of the Non-determinism of ChatGPT in Code Generation
• Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools
• "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
• Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
• Perfection Not Required? Human-AI Partnerships in Code Translation
• How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models
• Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
• Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting
• Better Together? An Evaluation of AI-Supported Code Translation
• Toward General Design Principles for Generative AI Applications
• Exploring Perspectives on the Impact of Artificial Intelligence on the Creativity of Knowledge Work: Beyond Mechanised Plagiarism and Stochastic Parrots
• What is it like to program with artificial intelligence?
• GitHub Copilot AI pair programmer: Asset or Liability?
• GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information
• AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
• Rethinking Explainability as a Dialogue: A Practitioner's Perspective
• Should Computers Be Easy To Use? Questioning the Doctrine of Simplicity in User Interface Design
• A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
• Next Steps for Human-Centered Generative AI: A Technical Perspective
• Grimm in Wonderland: Prompt Engineering with Midjourney to Illustrate Fairytales
• Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions
• Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordances
• Machine Explanations and Human Understanding
• The Power of Nudging: Exploring Three Interventions for Metacognitive Skills Instruction across Intelligent Tutoring Systems
• What the DAAM: Interpreting Stable Diffusion Using Cross Attention
• How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
• Quality Estimation & Interpretability for Code Translation

Cited by in corpus (24)

• Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
• WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization
• AI Should Challenge, Not Obey
• Generative AI in Knowledge Work: Design Implications for Data Navigation and Decision-Making
• The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks
• Are We On Track? AI-Assisted Active and Passive Goal Reflection During Meetings
• Collage is the New Writing: Exploring the Fragmentation of Text and User Interfaces in AI Tools
• VeriPlan: Integrating Formal Verification and LLMs into End-User Planning
• User Experience with LLM-powered Conversational Recommendation Systems: A Case of Music Recommendation
• Rethinking Citation of AI Sources in Student-AI Collaboration within HCI Design Education
• MeetMap: Real-Time Collaborative Dialogue Mapping with LLMs in Online Meetings
• How Scientists Use Large Language Models to Program
• From Following to Understanding: Investigating the Role of Reflective Prompts in AR-Guided Tasks to Promote Task Understanding
• NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
• What Does Success Look Like? Catalyzing Meeting Intentionality with AI-Assisted Prospective Reflection
• Judgment of Learning: A Human Ability Beyond Generative Artificial Intelligence
• Composable Prompting Workspaces for Creative Writing: Exploration and Iteration Using Dynamic Widgets
• OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models
• Inclusive Emotion Technologies: Addressing the Needs of d/Deaf and Hard of Hearing Learners in Video-Based Learning
• Lessons for GenAI Literacy From a Field Study of Human-GenAI Augmentation in the Workplace
• Computational Hermeneutics: Evaluating generative AI as a cultural technology
• CorpusStudio: Surfacing Emergent Patterns in a Corpus of Prior Work while Writing
• Content-Driven Local Response: Supporting Sentence-Level and Message-Level Mobile Email Replies With and Without AI
• OSINT Clinic: Co-designing AI-Augmented Collaborative OSINT Investigations for Vulnerability Assessment