Distributionally Robust Weighted -Nearest Neighbors
arXiv:2006.04004
Abstract
Learning a robust classifier from a few samples remains a key challenge in machine learning. A major thrust of research has been focused on developing -nearest neighbor (-NN) based algorithms combined with metric learning that captures similarities between samples. When the samples are limited, robustness is especially crucial to ensure the generalization capability of the classifier. In this paper, we study a minimax distributionally robust formulation of weighted -nearest neighbors, which aims to find the optimal weighted -NN classifiers that hedge against feature uncertainties. We develop an algorithm, \texttt{Dr.k-NN}, that efficiently solves this functional optimization problem and features in assigning minimax optimal weights to training samples when performing classification. These weights are class-dependent, and are determined by the similarities of sample features under the least favorable scenarios. When the size of the uncertainty set is properly tuned, the robust classifier has a smaller Lipschitz norm than the vanilla -NN, and thus improves the generalization capability. We also couple our framework with neural-network-based feature embedding. We demonstrate the competitive performance of our algorithm compared to the state-of-the-art in the few-training-sample setting with various real-data experiments.
References in corpus (7)
- Prototypical Networks for Few-shot Learning
- Certifying Some Distributional Robustness with Principled Adversarial Training
- Robust Wasserstein Profile Inference and Applications to Machine Learning
- Distributionally Robust Logistic Regression
- Differentiable Convex Optimization Layers
- Regularization via Mass Transportation
- Robust Hypothesis Testing Using Wasserstein Uncertainty Sets