Distributionally Robust Optimization and Generalization in Kernel Methods
arXiv:1905.10943
Abstract
Distributionally robust optimization (DRO) has attracted attention in machine learning due to its connections to regularization, generalization, and robustness. Existing work has considered uncertainty sets based on phi-divergences and Wasserstein distances, each of which have drawbacks. In this paper, we study DRO with uncertainty sets measured via maximum mean discrepancy (MMD). We show that MMD DRO is roughly equivalent to regularization by the Hilbert norm and, as a byproduct, reveal deep connections to classic results in statistical learning. In particular, we obtain an alternative proof of a generalization bound for Gaussian kernel ridge regression via a DRO lense. The proof also suggests a new regularizer. Our results apply beyond kernel methods: we derive a generically applicable approximation of MMD DRO, and show that it generalizes recent work on variance-based regularization.
References in corpus (10)
- Generative Moment Matching Networks
- Robustness and Regularization of Support Vector Machines
- Robust Wasserstein Profile Inference and Applications to Machine Learning
- Empirical Bernstein Bounds and Sample Variance Penalization
- Training generative neural networks via Maximum Mean Discrepancy optimization
- Distributionally Robust Logistic Regression
- Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
- A Kernel Test of Goodness of Fit
- A Kernel Perspective for Regularizing Deep Neural Networks
- Optimal Transport Based Distributionally Robust Optimization: Structural Properties and Iterative Schemes