6 papers · 1 filter
On Defining Erasure Harms for NLP
Yu Lu Liu, Arnav Goel, Jackie Chi Kit Cheung +3
The deployment of NLP systems has raised concerns about harms they might produce, including representational harms. Recent literature has begun to conceptualize and measure one suc…
Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation
Katelyn Xiaoying Mei, Yi-Li Hsu, Minjoon Choi +5
Human evaluation plays a critical role in assessing the quality of generated text. However, the reliability and reproducibility of these evaluations depend on transparent and well-…
Dehumanizing Machines: Mitigating Anthropomorphic Behaviors in Text Generation Systems
Myra Cheng, Su Lin Blodgett, Alicia DeVrio +2
As text generation systems' outputs are increasingly anthropomorphic -- perceived as human-like -- scholars have also increasingly raised concerns about how such outputs can lead t…
PERSONA: A Reproducible Testbed for Pluralistic Alignment
Louis Castricato, Nathan Lile, Rafael Rafailov +2
The rapid advancement of language models (LMs) necessitates robust alignment with diverse user values. However, current preference optimization approaches often fail to capture the…
ECBD: Evidence-Centered Benchmark Design for NLP
Yu Lu Liu, Su Lin Blodgett, Jackie Chi Kit Cheung +3
Benchmarking is seen as critical to assessing progress in NLP. However, creating a benchmark involves many design decisions (e.g., which datasets to include, which metrics to use)…
What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models
Jeongrok Yu, Seong Ug Kim, Jacob Choi +1
Bias is a disproportionate prejudice in favor of one side against another. Due to the success of transformer-based Masked Language Models (MLMs) and their impact on many NLP tasks,…