FairPair: A Robust Evaluation of Biases in Language Models through Paired Perturbations

Jane Dwivedi-Yu, Raaz Dwivedi, Timo Schick

The accurate evaluation of differential treatment in language models to specific groups is critical to ensuring a positive and safe user experience. An ideal evaluation should have…