When Evaluators Cry Wolf: Lessons from Production LLM-as-Judge Evaluation in Educational AI

Chris Rohlfs, Rodrigo Vergara Bosse, Priscilla Rain Hopper +2

MagicSchool's K-12 AI product suite is used by millions of teachers and serves millions of teacher and student messages each month. Our team monitors output along four priority dim…