16 papers
Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue
Clayton Cohn, Joyce Fonteles, Kirk Vanacore +5
In K-12 mathematics tutoring, student-tutor dialogue provides rich evidence of learners' problem-solving processes and sources of difficulty. Learning analytics research increasing…
Examining Variation in How Guided AI Tutors Resolve Student Impasses
Bakhtawar Ahtisham, Kirk Vanacore, Alessandra Napoli +5
When a student is stuck, a tutor faces the assistance dilemma: help given too early can hinder productive struggle, while help withheld too long leaves the student in a frustrating…
Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Zeyu He, Zhuqian Zhou, Kirk Vanacore +2
Large-scale text annotation brings expert insight to millions of documents, often through a codebook that AI annotators follow. Developing a robust codebook, however, takes months.…
Think Before You Accept: Can Written Justification Reduce Uncritical Uptake of AI Writing Suggestions?
Yan Tao, Jennifer Meyer, Rene F. Kizilcec
Generative AI can offer students useful feedback, but its value depends on judging which suggestions are accurate and relevant. Prior research shows that strategic friction during…
Redact or Keep? A Fully Local AI Cascade for Educational Dialogue De-Identification
Haocheng Zhang, Zhuqian Zhou, Kirk Vanacore +2
Educational dialogue is a valuable but sensitive resource for research: the same transcripts that capture authentic learning often capture personally identifiable information (PII)…
Who Decides in AI-Mediated Learning? The Agency Allocation Framework
Conrad Borchers, Olga Viberg, René F. Kizilcec
As AI-mediated learning systems increasingly shape how learners plan, make decisions, and progress through education, learner agency is becoming both more consequential and harder…
Teachers' Perceived Benefits and Risks of AI Across Fifty-Five Countries: An Audit of LLM Alignment and Steerability
Yan Tao, Olga Viberg, Deepak Varuvel Dennison +2
Teachers' trust in artificial intelligence (AI) in education depends on how they balance its perceived benefits and risks. Yet global discussions about scaling AI in education rely…
Does the TalkMoves Codebook Generalize to One-on-One Tutoring and Multimodal Interaction?
Corina Luca Focsan, Marie Cynthia Abijuru Kamikazi, Tamisha Thompson +5
Accountable Talk theory has been widely adopted to analyze classroom discourse and is increasingly used to annotate tutoring interactions. In particular, the TalkMoves codebook, gr…
Million Tutoring Moves (MTM): An Open Multimodal Dataset for the Science of Tutoring
René Kizilcec, Kirk Vanacore, Zhuqian Zhou +8
We introduce the Million Tutoring Moves (MTM) project, an open dataset initiative aimed at advancing the science of tutoring through large-scale, reusable, and multimodal interacti…
Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts
Jinsook Lee, Kirk Vanacore, Zhuqian Zhou +2
Automated annotation of pedagogical dialogue is a high-stakes task where LLMs often fail without sufficient domain grounding. We present a domain-adapted RAG pipeline for tutoring…
Modernizing Ground Truth: Four Shifts Toward Improving Reliability and Validity in AI in Education
Danielle R. Thomas, Conrad Borchers, Kirk P. Vanacore +2
Generative Artificial Intelligence (GenAI) is now widespread in education, yet the efficacy of GenAI systems remains constrained by the quality and interpretation of the labeled da…
The Future of Feedback: How Can AI Help Transform Feedback to Be More Engaging, Effective, and Scalable?
Jennifer Meyer, Olaf Köller, Thorben Jansen +33
With digital learning environments becoming more prevalent, the ease with which generative AI enables the scalable production of real-time, automated feedback holds the potential t…
Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration
Bakhtawar Ahtisham, Kirk Vanacore, Rene F. Kizilcec
Large language models (LLMs) are increasingly positioned as scalable tools for annotating educational data, including classroom discourse, interaction logs, and qualitative learnin…
Tutor Move Taxonomy: A Theory-Aligned Framework for Analyzing Instructional Moves in Tutoring
Zhuqian Zhou, Kirk Vanacore, Tamisha Thompson +2
Understanding what makes tutoring effective requires methods for systematically analyzing tutors' instructional actions during learning interactions. This paper presents a tutor mo…
Protecting and Promoting Human Agency in Education in the Age of Artificial Intelligence
Olga Viberg, Mutlu Cukurova, Rene F. Kizilcec +12
Human agency is crucial in education and increasingly challenged by the use of generative AI. This meeting report synthesizes interdisciplinary insights and conceptualizes four asp…
A Causal Framework for Estimating Heterogeneous Effects of On-Demand Tutoring
Kirk Vanacore, Danielle R Thomas, Digory Smith +3
This paper introduces a scalable causal inference framework for estimating the immediate, session-level effects of on-demand human tutoring embedded within adaptive learning system…