121 citations · 181 across the 19 of their papers we have counts for
23 papers
Pigeonholing: how bad prompts hurt models, causing collapse and mistakes
Hyunji Nam, Keertana Chidambaram, Dorottya Demszky +1
While in-context learning is generally shown to be effective in Large Language Models (LLMs), bad contexts can cause performance degradation and mode collapse, a phenomenon we call…
From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment
Ivo Bueno, Babette Bühler, Philipp Stark +5
Automated scoring models are increasingly used to assign rubric-based quality ratings to complex language performances, including classroom transcripts, yet they typically provide…
Mitigating LLM biases toward spurious social contexts using direct preference optimization
Hyunji Nam, Dorottya Demszky
LLMs are increasingly used for high-stakes decision-making, yet their sensitivity to spurious context can introduce harmful biases. This is a critical concern when models are deplo…
Practitioner Voices Summit: How Teachers Evaluate AI Tools through Deliberative Sensemaking
Dorottya Demszky, Christopher Mah, Helen Higgins
Teachers face growing pressure to integrate AI tools into their classrooms, yet are rarely positioned as agentic decision-makers in this process. Understanding the criteria teacher…
Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback
Mei Tan, Lena Phalen, Dorottya Demszky
Effective personalized feedback is critical to students' literacy development. Though LLM-powered tools now promise to automate such feedback at scale, LLMs are not language-neutra…
DiagramIR: An Automatic Pipeline for Educational Math Diagram Evaluation
Vishal Kumar, Shubhra Mishra, Rebecca Hao +3
Large Language Models (LLMs) are increasingly being adopted as tools for learning; however, most tools remain text-only, limiting their usefulness for domains where visualizations…
IDEAlign: Comparing Ideas of Large Language Models to Domain Expert
Hyunji Nam, Lucia Langlois, James Malamut +2
Large language models (LLMs) are increasingly used to produce open-ended, interpretive annotations, yet there is no validated, scalable measure of idea-level similarity to expert a…
EduCoder: An Open-Source Annotation System for Education Transcript Data
Saad Ashraf, James Malamut, Vishal Kumar +7
We introduce EduCoder, a domain-specialized tool designed to support utterance-level annotation of educational dialogue. While general-purpose text annotation tools for NLP and qua…
From Weak Labels to Strong Results: Utilizing 5,000 Hours of Noisy Classroom Transcripts with Minimal Accurate Data
Ahmed Adel Attia, Dorottya Demszky, Jing Liu +1
Recent progress in speech recognition has relied on models trained on vast amounts of labeled data. However, classroom Automatic Speech Recognition (ASR) faces the real-world chall…
Continued Pretraining for Domain Adaptation of Wav2vec2.0 in Automatic Speech Recognition for Elementary Math Classroom Settings
Ahmed Adel Attia, Dorottya Demszky, Tolulope Ogunremi +2
Creating Automatic Speech Recognition (ASR) systems that are robust and resilient to classroom conditions is paramount to the development of AI tools to aid teachers and students.…
Measuring Five Accountable Talk Moves to Improve Instruction at Scale
Ashlee Kupor, Candice Morgan, Dorottya Demszky
Providing consistent, individualized feedback to teachers on their instruction can improve student learning outcomes. Such feedback can especially benefit novice instructors who te…
Kid-Whisper: Towards Bridging the Performance Gap in Automatic Speech Recognition for Children VS. Adults
Ahmed Adel Attia, Jing Liu, Wei Ai +2
Recent advancements in Automatic Speech Recognition (ASR) systems, exemplified by Whisper, have demonstrated the potential of these systems to approach human-level performance give…
SIGHT: A Large Annotated Dataset on Student Insights Gathered from Higher Education Transcripts
Rose E. Wang, Pawan Wirawarn, Noah Goodman +1
Lectures are a learning experience for both students and teachers. Students learn from teachers about the subject material, while teachers learn from students about how to refine t…
Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction
Rose E. Wang, Dorottya Demszky
Coaching, which involves classroom observation and expert feedback, is a widespread and fundamental part of teacher training. However, the majority of teachers do not have access t…
MD3: The Multi-Dialect Dataset of Dialogues
Jacob Eisenstein, Vinodkumar Prabhakaran, Clara Rivera +2
We introduce a new dataset of conversational speech representing English from India, Nigeria, and the United States. The Multi-Dialect Dataset of Dialogues (MD3) strikes a new bala…
The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts
Dorottya Demszky, Heather Hill
Classroom discourse is a core medium of instruction - analyzing it can provide a window into teaching and learning as well as driving the development of new tools for improving ins…
Computationally Identifying Funneling and Focusing Questions in Classroom Discourse
Sterling Alic, Dorottya Demszky, Zid Mancenido +3
Responsive teaching is a highly effective strategy that promotes student learning. In math classrooms, teachers might "funnel" students towards a normative answer or "focus" studen…
Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions
Dorottya Demszky, Jing Liu, Zid Mancenido +4
In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said. In e…
Learning to Recognize Dialect Features
Dorottya Demszky, Devyani Sharma, Jonathan H. Clark +2
Building NLP systems that serve everyone requires accounting for dialect differences. But dialects are not monolithic entities: rather, distinctions between and within dialects are…
The Role of Verb Semantics in Hungarian Verb-Object Order
Dorottya Demszky, László Kálmán, Dan Jurafsky +1
Hungarian is often referred to as a discourse-configurational language, since the structural position of constituents is determined by their logical function (topic or comment) rat…
GoEmotions: A Dataset of Fine-Grained Emotions
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko +3
Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior. Advancement in this area can b…
Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings
Dorottya Demszky, Nikhil Garg, Rob Voigt +4
We provide an NLP framework to uncover four linguistic dimensions of political polarization in social media: topic choice, framing, affect and illocutionary force. We quantify thes…
Transforming Question Answering Datasets Into Natural Language Inference Datasets
Dorottya Demszky, Kelvin Guu, Percy Liang
Existing datasets for natural language inference (NLI) have propelled research on language understanding. We propose a new method for automatically deriving NLI datasets from the g…