Incorporating Unlabeled Data into Distributionally Robust Learning

Abstract

We study a robust alternative to empirical risk minimization called distributionally robust learning (DRL), in which one learns to perform against an adversary who can choose the data distribution from a specified set of distributions. We illustrate a problem with current DRL formulations, which rely on an overly broad definition of allowed distributions for the adversary, leading to learned classifiers that are unable to predict with any confidence. We propose a solution that incorporates unlabeled data into the DRL problem to further constrain the adversary. We show that this new formulation is tractable for stochastic gradient-based optimization and yields a computable guarantee on the future performance of the learned classifier, analogous to -- but tighter than -- guarantees from conventional DRL. We examine the performance of this new formulation on 14 real datasets and find that it often yields effective classifiers with nontrivial performance guarantees in situations where conventional DRL produces neither. Inspired by these results, we extend our DRL formulation to active learning with a novel, distributionally-robust version of the standard model-change heuristic. Our active learning algorithm often achieves superior learning performance to the original heuristic on real datasets.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Intriguing properties of neural networks
• End to End Learning for Self-Driving Cars
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Robust Physical-World Attacks on Deep Learning Models
• An Empirical Evaluation of Deep Learning on Highway Driving
• Distributional Smoothing with Virtual Adversarial Training
• Robust Wasserstein Profile Inference and Applications to Machine Learning
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Semidefinite relaxations for certifying robustness to adversarial examples
• Distributionally Robust Logistic Regression
• A Benchmark and Comparison of Active Learning for Logistic Regression

Cited by in corpus (4)

• Towards Out-Of-Distribution Generalization: A Survey
• A Distributionally Robust Approach to Fair Classification
• Kernelized Heterogeneous Risk Minimization
• Distributionally Robust Learning with Stable Adversarial Training