6 papers · 1 filter
Characterizing Language Use in a Collaborative Situated Game
Nicholas Tomlin, Naitian Zhou, Eve Fleisig +10
Cooperative video games, where multiple participants must coordinate by communicating and reasoning under uncertainty in complex environments, yield a rich source of language data.…
Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions
Eve Fleisig, Matthias Orlikowski, Philipp Cimiano +1
For machine learning datasets to accurately represent diverse opinions in a population, they must preserve variation in data labels while filtering out spam or low-quality response…
Accurate and Data-Efficient Toxicity Prediction when Annotators Disagree
Harbani Jaggi, Kashyap Murali, Eve Fleisig +1
When annotators disagree, predicting the labels given by individual annotators can capture nuances overlooked by traditional label aggregation. We introduce three approaches to pre…
Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination
Eve Fleisig, Genevieve Smith, Madeline Bossi +3
We present a large-scale study of linguistic bias exhibited by ChatGPT covering ten dialects of English (Standard American English, Standard British English, and eight widely spoke…
Standard Language Ideology in AI-Generated Language
Genevieve Smith, Eve Fleisig, Ishita Rustagi +1
Large language models (LLMs) generate text that reinforces standard language ideology: a bias towards certain language varieties that are granted more prestige, authority, and legi…
Incorporating Worker Perspectives into MTurk Annotation Practices for NLP
Olivia Huang, Eve Fleisig, Dan Klein
Current practices regarding data collection for natural language processing on Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data quality and heuristics…