Robust Wasserstein Profile Inference and Applications to Machine Learning

Abstract

We show that several machine learning estimators, including square-root LASSO (Least Absolute Shrinkage and Selection) and regularized logistic regression can be represented as solutions to distributionally robust optimization (DRO) problems. The associated uncertainty regions are based on suitably defined Wasserstein distances. Hence, our representations allow us to view regularization as a result of introducing an artificial adversary that perturbs the empirical distribution to account for out-of-sample effects in loss estimation. In addition, we introduce RWPI (Robust Wasserstein Profile Inference), a novel inference methodology which extends the use of methods inspired by Empirical Likelihood to the setting of optimal transport costs (of which Wasserstein distances are a particular case). We use RWPI to show how to optimally select the size of uncertainty regions, and as a consequence, we are able to choose regularization parameters for these machine learning estimators without the use of cross validation. Numerical experiments are also given to validate our theoretical findings.

References in corpus (8)

• Robust Wasserstein Profile Inference and Applications to Machine Learning
• Extending the scope of empirical likelihood
• Regularization via Mass Transportation
• Sample Out-Of-Sample Inference Based on Wasserstein Distance
• Optimal Transport Based Distributionally Robust Optimization: Structural Properties and Iterative Schemes
• Semi-supervised Learning based on Distributionally Robust Optimization
• LASSO, Iterative Feature Selection and the Correlation Selector: Oracle Inequalities and Numerical Performances
• The Empirical Likelihood Approach to Quantifying Uncertainty in Sample Average Approximation

Cited by in corpus (83)

• Certifying Some Distributional Robustness with Principled Adversarial Training
• Robust Wasserstein Profile Inference and Applications to Machine Learning
• Distributionally Robust Optimization: A Review
• Training individually fair ML models with Sensitive Subspace Robustness
• Robust Validation: Confident Predictions Even When Distributions Shift
• Large-Scale Methods for Distributionally Robust Optimization
• Robust Hypothesis Testing Using Wasserstein Uncertainty Sets
• Sample Out-Of-Sample Inference Based on Wasserstein Distance
• Optimal Transport Based Distributionally Robust Optimization: Structural Properties and Iterative Schemes
• On Distributionally Robust Chance Constrained Programs with Wasserstein Distance
• Sensitivity analysis of Wasserstein distributionally robust optimization problems
• Poisoning Attack against Estimating from Pairwise Comparisons
• An Optimal Transport Approach to Personalized Federated Learning
• A Distributionally Robust Approach to Fair Classification
• Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning
• Distributional Robustness and Regularization in Reinforcement Learning
• Distributionally Robust Optimization and Generalization in Kernel Methods
• Semi-supervised Learning based on Distributionally Robust Optimization
• Online Stochastic Optimization with Wasserstein Based Non-stationarity
• Kernel Distributionally Robust Optimization
• Data Heterogeneity Modeling for Trustworthy Machine Learning
• A First-Order Algorithmic Framework for Wasserstein Distributionally Robust Logistic Regression
• Incorporating Unlabeled Data into Distributionally Robust Learning
• Multivariate Distributionally Robust Convex Regression under Absolute Error Loss
• Robustifying Conditional Portfolio Decisions via Optimal Transport
• Machine Learning's Dropout Training is Distributionally Robust Optimal
• Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curation
• Solving Multistage Stochastic Linear Programming via Regularized Linear Decision Rules: An Application to Hydrothermal Dispatch Planning
• Tractable Reformulations of Distributionally Robust Two-stage Stochastic Programs with $\infty -$Wasserstein Distance
• Gromov-Wasserstein Distances between Gaussian Distributions
• Doubly Robust Data-Driven Distributionally Robust Optimization
• Sinkhorn Distributionally Robust Optimization
• The Geometry of Adversarial Training in Binary Classification
• Model Calibration via Distributionally Robust Optimization: On the NASA Langley Uncertainty Quantification Challenge
• Fast Epigraphical Projection-based Incremental Algorithms for Wasserstein Distributionally Robust Support Vector Machine
• Lipschitz Networks and Distributional Robustness
• Regularization Helps with Mitigating Poisoning Attacks: Distributionally-Robust Machine Learning Using the Wasserstein Distance
• Optimization-based Quantification of Simulation Input Uncertainty via Empirical Likelihood
• Efficient Stochastic Gradient Descent for Learning with Distributionally Robust Optimization
• Combating Conservativeness in Data-Driven Optimization under Uncertainty: A Solution Path Approach
• Adversarial Classification: Necessary conditions and geometric flows
• Generalised Lipschitz Regularisation Equals Distributional Robustness
• Distributionally Robust Bottleneck Combinatorial Problems: Uncertainty Quantification and Robust Decision Making
• Robust Fairness-aware Learning Under Sample Selection Bias
• Robust Arbitrage Conditions for Financial Markets
• Distributionally Robust Receive Combining
• Robustified Multivariate Regression and Classification Using Distributionally Robust Optimization under the Wasserstein Metric
• Towards Optimal Problem Dependent Generalization Error Bounds in Statistical Learning Theory
• Distributionally Robust XVA via Wasserstein Distance: Wrong Way Counterparty Credit and Funding Risk
• Bounding Optimality Gap in Stochastic Optimization via Bagging: Statistical Efficiency and Stability
• Distributionally Robust Formulation and Model Selection for the Graphical Lasso
• Distributionally robust halfspace depth
• Optimal Transport Relaxations with Application to Wasserstein GANs
• Distributionally Robust Weighted $k$-Nearest Neighbors
• Variance Reduction via Primal-Dual Accelerated Dual Averaging for Nonsmooth Convex Finite-Sums
• Optimization-based Calibration of Simulation Input Models
• Distributional Robustness with IPMs and links to Regularization and GANs
• Family-wise error rate control in Gaussian graphical model selection via Distributionally Robust Optimization
• Scalable Algorithms for the Sparse Ridge Regression
• Fast and Robust Rank Aggregation against Model Misspecification
• A Law of Robustness for Weight-bounded Neural Networks
• Adversarially Robust Kernel Smoothing
• Distributionally Robust Martingale Optimal Transport
• Distributionally Robust Profit Opportunities
• Principled learning method for Wasserstein distributionally robust optimization with local perturbations
• Distributional Robustness Regularized Scenario Optimization with Application to Model Predictive Control
• Formulation and properties of a divergence used to compare probability measures without absolute continuity and its application to uncertainty quantification
• A Distributionally Robust Optimization Method for Adversarial Multiple Kernel Learning
• Distributional Robust Kelly Gambling: Optimal Strategy under Uncertainty in the Long-Run
• Unbiased Gradient Estimation for Distributionally Robust Learning
• Learning Against Distributional Uncertainty: On the Trade-off Between Robustness and Specificity
• On Regularized Square-root Regression Problems: Distributionally Robust Interpretation and Fast Computations
• Data-driven two-stage conic optimization with zero-one uncertainties
• A Note on Piecewise Affine Decision Rules for Robust, Stochastic, and Data-Driven Optimization
• Parametric Scenario Optimization under Limited Data: A Distributionally Robust Optimization View
• Efficient Data-Driven Optimization with Noisy Data
• Individually Fair Gradient Boosting
• Orthounimodal Distributionally Robust Optimization: Representation, Computation and Multivariate Extreme Event Applications
• Human Imperceptible Attacks and Applications to Improve Fairness
• Minimax Optimal Estimation of Stability Under Distribution Shift
• Robust Learning in Heterogeneous Contexts
• Distributionally Robust Newsvendor with Moment Constraints
• Higher-Order Expansion and Bartlett Correctability of Distributionally Robust Optimization