Distributionally Robust Logistic Regression

Abstract

This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown data-generating distribution with high confidence. We then formulate a distributionally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

Neural Information Processing Systems (NIPS), 2015

Cited by in corpus (82)

• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Robust Wasserstein Profile Inference and Applications to Machine Learning
• Towards Out-Of-Distribution Generalization: A Survey
• Conic Programming Reformulations of Two-Stage Distributionally Robust Linear Programs over Wasserstein Balls
• Distributionally Robust Optimization: A Review
• On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
• Training individually fair ML models with Sensitive Subspace Robustness
• Towards a Theoretical Framework of Out-of-Distribution Generalization
• Robust Hypothesis Testing Using Wasserstein Uncertainty Sets
• It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks
• Data-driven distributionally robust MPC for constrained stochastic systems
• Sample Out-Of-Sample Inference Based on Wasserstein Distance
• Wasserstein Distributionally Robust Kalman Filtering
• Recent theoretical advances in decentralized distributed convex optimization
• Optimal Transport Based Distributionally Robust Optimization: Structural Properties and Iterative Schemes
• On Distributionally Robust Chance Constrained Programs with Wasserstein Distance
• Evaluating Model Robustness and Stability to Dataset Shift
• Decomposition Algorithm for Distributionally Robust Optimization using Wasserstein Metric
• A Distributionally Robust Approach to Fair Classification
• Ensuring Fairness Beyond the Training Data
• Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning
• Distributional Robustness and Regularization in Reinforcement Learning
• Distributionally Robust Optimization and Generalization in Kernel Methods
• A Distributionally Robust Area Under Curve Maximization Model
• Semi-supervised Learning based on Distributionally Robust Optimization
• Distributed Saddle-Point Problems Under Similarity
• Multivariate Distributionally Robust Convex Regression under Absolute Error Loss
• A First-Order Algorithmic Framework for Wasserstein Distributionally Robust Logistic Regression
• Incorporating Unlabeled Data into Distributionally Robust Learning
• Regularized Optimal Transport is Ground Cost Adversarial
• Data-driven Optimal Cost Selection for Distributionally Robust Optimization
• Doubly Robust Data-Driven Distributionally Robust Optimization
• Sinkhorn Distributionally Robust Optimization
• Fast Distributionally Robust Learning with Variance Reduced Min-Max Optimization
• Twice regularized MDPs and the equivalence between robustness and regularization
• A Distributionally Robust Boosting Algorithm
• Accounting for Unobserved Confounding in Domain Generalization
• Minimax Classification with 0-1 Loss and Performance Guarantees
• Online Stochastic Convex Optimization: Wasserstein Distance Variation
• Risk Variance Penalization
• Distributional Robustness Loss for Long-tail Learning
• A Stochastic Subgradient Method for Distributionally Robust Non-Convex Learning
• Why do classifier accuracies show linear trends under distribution shift?
• Data-driven Inverse Optimization with Imperfect Information
• Understanding the effect of sparsity on neural networks robustness
• Distributionally Robust Formulation and Model Selection for the Graphical Lasso
• Near-Optimal Decentralized Algorithms for Saddle Point Problems over Time-Varying Networks
• Regularity as Regularization: Smooth and Strongly Convex Brenier Potentials in Optimal Transport
• Distributionally Robust Parametric Maximum Likelihood Estimation
• Optimization-based Calibration of Simulation Input Models
• Coordinate Linear Variance Reduction for Generalized Linear Programming
• Distributionally Robust Weighted $k$-Nearest Neighbors
• Wasserstein Distributionally Robust Inverse Multiobjective Optimization
• Robust GANs against Dishonest Adversaries
• Optimal Transport Relaxations with Application to Wasserstein GANs
• Distributional Robustness with IPMs and links to Regularization and GANs
• Robust Graph Learning Under Wasserstein Uncertainty
• Supervised classification via minimax probabilistic transformations
• Fast and Robust Rank Aggregation against Model Misspecification
• On Regularized Square-root Regression Problems: Distributionally Robust Interpretation and Fast Computations
• Distributionally Robust Prescriptive Analytics with Wasserstein Distance
• Robust Generalization despite Distribution Shift via Minimum Discriminating Information
• A Distributionally Robust Optimization Method for Adversarial Multiple Kernel Learning
• Distributional Robust Kelly Gambling: Optimal Strategy under Uncertainty in the Long-Run
• A Robust Learning Algorithm for Regression Models Using Distributionally Robust Optimization under the Wasserstein Metric
• Distributionally Robust Multi-Output Regression Ranking
• Randomized Stochastic Gradient Descent Ascent
• Adversarially Robust Kernel Smoothing
• Principled learning method for Wasserstein distributionally robust optimization with local perturbations
• Stochastic Projective Splitting: Solving Saddle-Point Problems with Multiple Regularizers
• Zeroth-Order Methods for Convex-Concave Minmax Problems: Applications to Decision-Dependent Risk Minimization
• Robust Learning in Heterogeneous Contexts
• Correlation Robust Influence Maximization
• Quasi-Newton Methods for Saddle Point Problems and Beyond
• Human Imperceptible Attacks and Applications to Improve Fairness
• Group-Structured Adversarial Training
• Distributionally Robust Learning with Stable Adversarial Training
• Understanding Distributional Ambiguity via Non-robust Chance Constraint
• Distributionally Robust Graphical Models
• Causality and Generalizability: Identifiability and Learning Methods
• Certifying Robustness to Programmable Data Bias in Decision Trees