Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions

Abstract

Low-dimensional embedding, manifold learning, clustering, classification, and anomaly detection are among the most important problems in machine learning. The existing methods usually consider the case when each instance has a fixed, finite-dimensional feature representation. Here we consider a different setting. We assume that each instance corresponds to a continuous probability distribution. These distributions are unknown, but we are given some i.i.d. samples from each distribution. Our goal is to estimate the distances between these distributions and use these distances to perform low-dimensional embedding, clustering/classification, or anomaly detection for the distributions. We present estimation algorithms, describe how to apply them for machine learning tasks on distributions, and show empirical results on synthetic data, real word images, and astronomical data sets.

References in corpus (6)

• Probabilistic Latent Semantic Analysis
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• A class of Rényi information estimators for multidimensional densities
• Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs
• Anomaly Detection with Score functions based on Nearest Neighbor Graphs
• Empirical estimation of entropy functionals with confidence

Cited by in corpus (22)

• Kernel Mean Embedding of Distributions: A Review and Beyond
• Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks
• One-Class Support Measure Machines for Group Anomaly Detection
• Likelihood Non-Gaussianity in Large-Scale Structure Analyses
• Learning Theory for Distribution Regression
• FairTest: Discovering Unwarranted Associations in Data-Driven Applications
• Kernels on Sample Sets via Nonparametric Divergence Estimates
• Direct estimation of density functionals using a polynomial basis
• Influence Functions for Machine Learning: Nonparametric Estimators for Entropies, Divergences and Mutual Informations
• Estimating Mutual Information by Local Gaussian Approximation
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality
• Exponential Series Approaches for Nonparametric Graphical Models
• Learning to Benchmark: Determining Best Achievable Misclassification Error from Training Data
• Variational Representations and Neural Network Estimation of Rényi Divergences
• Distributional Robustness and Uncertainty Quantification for Rare Events
• On the Impossibility of Statistically Improving Empirical Optimization: A Second-Order Stochastic Dominance Perspective
• Linear-Complexity Exponentially-Consistent Tests for Universal Outlying Sequence Detection
• Learned Uncertainty Calibration for Visual Inertial Localization
• Neural Estimation of Statistical Divergences
• Non-Asymptotic Performance Guarantees for Neural Estimation of $\mathsf{f}$-Divergences
• Irregular Convolutional Auto-Encoder on Point Clouds
• co-BPM: a Bayesian Model for Divergence Estimation