A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification

Abstract

One of the pursued objectives of deep learning is to provide tools that learn abstract representations of reality from the observation of multiple contextual situations. More precisely, one wishes to extract disentangled representations which are (i) low dimensional and (ii) whose components are independent and correspond to concepts capturing the essence of the objects under consideration (Locatello et al., 2019b). One step towards this ambitious project consists in learning disentangled representations with respect to a predefined (sensitive) attribute, e.g., the gender or age of the writer. Perhaps one of the main application for such disentangled representations is fair classification. Existing methods extract the last layer of a neural network trained with a loss that is composed of a cross-entropy objective and a disentanglement regularizer. In this work, we adopt an information-theoretic view of this problem which motivates a novel family of regularizers that minimizes the mutual information between the latent representation and the sensitive attribute conditional to the target. The resulting set of losses, called CLINIC, is parameter free and thus, it is easier and faster to train. CLINIC losses are studied through extensive numerical experiments by training over 2k neural networks. We demonstrate that our methods offer a better disentanglement/accuracy trade-off than previous techniques, and generalize better than training with cross-entropy loss solely provided that the disentanglement task is not too constraining.

arXiv commentFindings AACL 2023

References in corpus (29)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Decoupled Weight Decay Regularization
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• On the Opportunities and Risks of Foundation Models
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning Representations by Maximizing Mutual Information Across Views
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Fast is better than free: Revisiting adversarial training
• MINE: Mutual Information Neural Estimation
• An efficient framework for learning sentence representations
• Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
• Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model
• On Formalizing Fairness in Prediction with Machine Learning
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Learning Anonymized Representations with Adversarial Neural Networks
• Functional Isolation Forest
• What are the best systems? New perspectives on NLP Benchmarking
• A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
• Contrastive Learning for Fair Representations
• When OT meets MoM: Robust estimation of Wasserstein Distance
• Rethinking InfoNCE: How Many Negative Samples Do You Need?
• Beam Search with Bidirectional Strategies for Neural Response Generation
• From the Token to the Review: A Hierarchical Multimodal approach to Opinion Mining
• The Area of the Convex Hull of Sampled Curves: a Robust Functional Statistical Depth Measure
• Hypothesis Transfer Learning with Surrogate Classification Losses: Generalization Bounds through Algorithmic Stability