GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction

Virginia K. Felkner, Jennifer A. Thompson, Jonathan May

Social biases in LLMs are usually measured via bias benchmark datasets. Current benchmarks have limitations in scope, grounding, quality, and human effort required. Previous work h…