Unbiased Gradient Estimation for Distributionally Robust Learning

Abstract

Seeking to improve model generalization, we consider a new approach based on distributionally robust learning (DRL) that applies stochastic gradient descent to the outer minimization problem. Our algorithm efficiently estimates the gradient of the inner maximization problem through multi-level Monte Carlo randomization. Leveraging theoretical results that shed light on why standard gradient estimators fail, we establish the optimal parameterization of the gradient estimators of our approach that balances a fundamental tradeoff between computation time and statistical variance. Numerical experiments demonstrate that our DRL approach yields significant benefits over previous work.

arXiv commentICML 2020, AISTATS 2021 submission

References in corpus (6)

• Certifying Some Distributional Robustness with Principled Adversarial Training
• Robust Wasserstein Profile Inference and Applications to Machine Learning
• Decomposition Algorithm for Distributionally Robust Optimization using Wasserstein Metric
• OpenML-Python: an extensible Python API for OpenML
• Unbiased Simulation for Optimizing Stochastic Function Compositions
• Efficient Stochastic Gradient Descent for Learning with Distributionally Robust Optimization