Understanding Unequal Gender Classification Accuracy from Face Images

Abstract

Recent work shows unequal performance of commercial face classification services in the gender classification task across intersectional groups defined by skin type and gender. Accuracy on dark-skinned females is significantly worse than on any other group. In this paper, we conduct several analyses to try to uncover the reason for this gap. The main finding, perhaps surprisingly, is that skin type is not the driver. This conclusion is reached via stability experiments that vary an image's skin type via color-theoretic methods, namely luminance mode-shift and optimal transport. A second suspect, hair length, is also shown not to be the driver via experiments on face images cropped to exclude the hair. Finally, using contrastive post-hoc explanation techniques for neural networks, we bring forth evidence suggesting that differences in lip, eye and cheek structure across ethnicity lead to the differences. Further, lip and eye makeup are seen as strong predictors for a female face, which is a troubling propagation of a gender stereotype.

References in corpus (4)

• Methods for Interpreting and Understanding Deep Neural Networks
• Invertible Conditional GANs for image editing
• Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives
• InclusiveFaceNet: Improving Face Attribute Detection with Race and Gender Diversity

Cited by in corpus (9)

• Gendered Differences in Face Recognition Accuracy Explained by Hairstyles, Makeup, and Facial Morphology
• Measuring Hidden Bias within Face Recognition via Racial Phenotypes
• Fair Classification with Noisy Protected Attributes: A Framework with Provable Guarantees
• Analysis of Manual and Automated Skin Tone Assignments for Face Recognition Applications
• Gender Slopes: Counterfactual Fairness for Computer Vision Models by Attribute Manipulation
• Facial Analysis Systems and Down Syndrome
• Fair Classification with Adversarial Perturbations
• Matched sample selection with GANs for mitigating attribute confounding
• Does Face Recognition Error Echo Gender Classification Error?