Maximum Margin Multiclass Nearest Neighbors

Abstract

We develop a general framework for margin-based multicategory classification in metric spaces. The basic work-horse is a margin-regularized version of the nearest-neighbor classifier. We prove generalization bounds that match the state of the art in sample size $n$ and significantly improve the dependence on the number of classes $k$. Our point of departure is a nearly Bayes-optimal finite-sample risk bound independent of $k$. Although $k$-free, this bound is unregularized and non-adaptive, which motivates our main result: Rademacher and scale-sensitive margin bounds with a logarithmic dependence on $k$. As the best previous risk estimates in this setting were of order $\sqrt{k}$, our bound is exponentially sharper. From the algorithmic standpoint, in doubling metric spaces our classifier may be trained on $n$ examples in $O(n^2\log n)$ time and evaluated on new points in $O(\log n)$ time.

References in corpus (2)

• Fast Construction of Nets in Low Dimensional Metrics, and Their Applications
• Multiclass learnability and the ERM principle

Cited by in corpus (5)

• Tight Risk Bounds for Multi-Class Margin Classifiers
• Nearest-Neighbor Sample Compression: Efficiency, Consistency, Infinite Dimensions
• Universal Bayes consistency in metric spaces
• Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space
• Classification in asymmetric spaces via sample compression