4 papers
Data Annotation as Measurement
Emma Harvey, Allison Koenecke, Rene F. Kizilcec
Modern AI systems depend on annotated data, but annotation is rarely treated as the act of measurement that it is. Instead, annotation quality is commonly reduced to agreement: if…
A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
Emma Harvey, Rene F. Kizilcec, Allison Koenecke
Increasingly, individuals who engage in online activities are expected to interact with large language model (LLM)-based chatbots. Prior work has shown that LLMs can display dialec…
"Don't Forget the Teachers": Towards an Educator-Centered Understanding of Harms from Large Language Models in Education
Emma Harvey, Allison Koenecke, Rene F. Kizilcec
Education technologies (edtech) are increasingly incorporating new features built on large language models (LLMs), with the goals of enriching the processes of teaching and learnin…
Implementing New Technology in Educational Systems
Scott Allen, Lisa Bardach, Jamie Jirout +4
Educators are more than workers within educational systems; they are stewards of educational systems. They must analyze student performance data, identify patterns that inform targ…