16 citations · 23 across the 9 of their papers we have counts for
10 papers
Characterizing Language Use in a Collaborative Situated Game
Nicholas Tomlin, Naitian Zhou, Eve Fleisig +10
Cooperative video games, where multiple participants must coordinate by communicating and reasoning under uncertainty in complex environments, yield a rich source of language data.…
Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions
Eve Fleisig, Matthias Orlikowski, Philipp Cimiano +1
For machine learning datasets to accurately represent diverse opinions in a population, they must preserve variation in data labels while filtering out spam or low-quality response…
Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree
Harbani Jaggi, Kashyap Murali, Eve Fleisig +1
When annotators disagree, predicting the labels given by individual annotators can capture nuances overlooked by traditional label aggregation. We introduce three approaches to pre…
Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination
Eve Fleisig, Genevieve Smith, Madeline Bossi +3
We present a large-scale study of linguistic bias exhibited by ChatGPT covering ten dialects of English (Standard American English, Standard British English, and eight widely spoke…
Standard Language Ideology in AI-Generated Language
Genevieve Smith, Eve Fleisig, Ishita Rustagi +1
Large language models (LLMs) generate text that reinforces standard language ideology: a bias towards certain language varieties that are granted more prestige, authority, and legi…
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Eve Fleisig, Su Lin Blodgett, Dan Klein +1
Longstanding data labeling practices in machine learning involve collecting and aggregating labels from multiple annotators. But what should we do when annotators disagree? Though…
Incorporating Worker Perspectives into MTurk Annotation Practices for NLP
Olivia Huang, Eve Fleisig, Dan Klein
Current practices regarding data collection for natural language processing on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data quality and heuristics…
Ghostbuster: Detecting Text Ghostwritten by Large Language Models
Vivek Verma, Eve Fleisig, Nicholas Tomlin +1
We introduce Ghostbuster, a state-of-the-art system for detecting AI-generated text. Our method works by passing documents through a series of weaker language models, running a str…
Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection
Vyoma Raman, Eve Fleisig, Dan Klein
The impact of AI models on marginalized communities has traditionally been measured by identifying performance differences between specified demographic subgroups. Though this appr…
When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks
Eve Fleisig, Rediet Abebe, Dan Klein
Though majority vote among annotators is typically used for ground truth labels in natural language processing, annotator disagreement in tasks such as hate speech detection may re…