Textual Entailment is not a Better Bias Metric than Token Probability

Virginia K. Felkner, Allison Lim, Jonathan May

Measurement of social bias in language models is typically by token probability (TP) metrics, which are broadly applicable but have been criticized for their distance from real-wor…