51 citations · 121 across the 32 of their papers we have counts for
17 papers · 1 filter
On Defining Erasure Harms for NLP
Yu Lu Liu, Arnav Goel, Jackie Chi Kit Cheung +3
The deployment of NLP systems has raised concerns about harms they might produce, including representational harms. Recent literature has begun to conceptualize and measure one suc…
Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation
Katelyn Xiaoying Mei, Yi-Li Hsu, Minjoon Choi +5
Human evaluation plays a critical role in assessing the quality of generated text. However, the reliability and reproducibility of these evaluations depend on transparent and well-…
Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
Myra Cheng, Su Lin Blodgett, Alicia DeVrio +2
As text generation systems' outputs are increasingly anthropomorphic -- perceived as human-like -- scholars have also increasingly raised concerns about how such outputs can lead t…
PERSONA: A Reproducible Testbed for Pluralistic Alignment
Louis Castricato, Nathan Lile, Rafael Rafailov +2
The rapid advancement of language models (LMs) necessitates robust alignment with diverse user values. However, current preference optimization approaches often fail to capture the…
ECBD: Evidence-Centered Benchmark Design for NLP
Yu Lu Liu, Su Lin Blodgett, Jackie Chi Kit Cheung +3
Benchmarking is seen as critical to assessing progress in NLP. However, creating a benchmark involves many design decisions (e.g., which datasets to include, which metrics to use)…
What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models
Jeongrok Yu, Seong Ug Kim, Jacob Choi +1
Bias is a disproportionate prejudice in favor of one side against another. Due to the success of transformer-based Masked Language Models (MLMs) and their impact on many NLP tasks,…
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Yu Lu Liu, Meng Cao, Su Lin Blodgett +3
AI and NLP publication venues have increasingly encouraged researchers to reflect on possible ethical considerations, adverse impacts, and other responsible AI issues their work mi…
"One-Size-Fits-All"? Examining Expectations around What Constitute "Fair" or "Good" NLG System Behaviors
Li Lucy, Su Lin Blodgett, Milad Shokouhi +2
Fairness-related assumptions about what constitute appropriate NLG system behaviors range from invariance, where systems are expected to behave identically for social groups, to ad…
This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language Models
Seraphina Goldfarb-Tarrant, Eddie Ungless, Esma Balkir +1
Bias research in NLP seeks to analyse models for social biases, thus helping NLP practitioners uncover, measure, and mitigate social harms. We analyse the body of work that uses pr…
It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance
Arjun Subramonian, Xingdi Yuan, Hal Daumé +1
Progress in NLP is increasingly measured through benchmarks; hence, contextualizing progress requires understanding when and why practitioners may disagree about the validity of be…
Examining Political Rhetoric with Epistemic Stance Detection
Ankita Gupta, Su Lin Blodgett, Justin H Gross +1
Participants in political discourse employ rhetorical strategies -- such as hedging, attributions, or denials -- to display varying degrees of belief commitments to claims proposed…
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler +3
There are many ways to express similar things in text, which makes evaluating natural language generation (NLG) systems difficult. Compounding this difficulty is the need to assess…
A Survey of Race, Racism, and Anti-Racism in NLP
Anjalie Field, Su Lin Blodgett, Zeerak Waseem +1
Despite inextricable ties between race and language, little work has considered race in NLP research and development. In this work, we survey 79 papers from the ACL anthology that…
How to Write a Bias Statement: Recommendations for Submissions to the Workshop on Gender Bias in NLP
Christian Hardmeier, Marta R. Costa-jussà, Kellie Webster +2
At the Workshop on Gender Bias in NLP (GeBNLP), we'd like to encourage authors to give explicit consideration to the wider aspects of bias and its social implications. For the 2020…
Language (Technology) is Power: A Critical Survey of "Bias" in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé +1
We survey 146 papers analyzing "bias" in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyz…
Monte Carlo Syntax Marginals for Exploring and Using Dependency Parses
Katherine A. Keith, Su Lin Blodgett, Brendan O'Connor
Dependency parsing research, which has made significant gains in recent years, typically focuses on improving the accuracy of single-tree predictions. However, ambiguity is inheren…
Demographic Dialectal Variation in Social Media: A Case Study of African-American English
Su Lin Blodgett, Lisa Green, Brendan O'Connor
Though dialectal language is increasingly abundant on social media, few resources exist for developing NLP tools to handle such language. We conduct a case study of dialectal langu…