Locating Demographic Bias at the Attention-Head Level in CLIP's Vision Encoder

Alaa Yasser, Kittipat Phunjanna, Marcos Escudero Viñolo +2

Standard fairness audits of foundation models quantify that a model is biased, but not where inside the network the bias resides. We propose a mechanistic fairness audit that combi…