Quantifying Social Biases Using Templates is Unreliable

Preethi Seshadri, Pouya Pezeshkpour, Sameer Singh

Recently, there has been an increase in efforts to understand how large language models (LLMs) propagate and amplify social biases. Several works have utilized templates for fairne…