A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification

Abstract

One of the pursued objectives of deep learning is to provide tools that learn abstract representations of reality from the observation of multiple contextual situations. More precisely, one wishes to extract disentangled representations which are (i) low dimensional and (ii) whose components are independent and correspond to concepts capturing the essence of the objects under consideration (Locatello et al., 2019b). One step towards this ambitious project consists in learning disentangled representations with respect to a predefined (sensitive) attribute, e.g., the gender or age of the writer. Perhaps one of the main application for such disentangled representations is fair classification. Existing methods extract the last layer of a neural network trained with a loss that is composed of a cross-entropy objective and a disentanglement regularizer. In this work, we adopt an information-theoretic view of this problem which motivates a novel family of regularizers that minimizes the mutual information between the latent representation and the sensitive attribute conditional to the target. The resulting set of losses, called CLINIC, is parameter free and thus, it is easier and faster to train. CLINIC losses are studied through extensive numerical experiments by training over 2k neural networks. We demonstrate that our methods offer a better disentanglement/accuracy trade-off than previous techniques, and generalize better than training with cross-entropy loss solely provided that the disentanglement task is not too constraining.

Findings AACL 2023

References in corpus (9)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Learning Representations by Maximizing Mutual Information Across Views
• Fast is better than free: Revisiting adversarial training
• Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
• A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
• Contrastive Learning for Fair Representations
• Rethinking InfoNCE: How Many Negative Samples Do You Need?
• Beam Search with Bidirectional Strategies for Neural Response Generation
• The Area of the Convex Hull of Sampled Curves: a Robust Functional Statistical Depth Measure