Estimating divergence functionals and the likelihood ratio by convex risk minimization

Abstract

We develop and analyze $M$-estimation methods for divergence functionals and the likelihood ratios of two probability distributions. Our method is based on a non-asymptotic variational characterization of $f$-divergences, which allows the problem of estimating divergences to be tackled via convex empirical risk optimization. The resulting estimators are simple to implement, requiring only the solution of standard convex programs. We present an analysis of consistency and convergence for these estimators. Given conditions only on the ratios of densities, we show that our estimators can achieve optimal minimax rates for the likelihood ratio and the divergence functionals in certain regimes. We derive an efficient optimization algorithm for computing our estimates, and illustrate their convergence behavior and practical viability by simulations.

28 pages

References in corpus (1)

• On surrogate loss functions and $f$-divergences

Cited by in corpus (214)

• Disentangling by Factorising
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• Change-Point Detection in Time-Series Data by Relative Density-Ratio Estimation
• Recent Advances in Autoencoder-Based Representation Learning
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• On Mutual Information Maximization for Representation Learning
• Stabilizing Training of Generative Adversarial Networks through Regularization
• Hilbert space embeddings and metrics on probability measures
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• Least Squares Generative Adversarial Networks
• Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
• Unified Approach to Classical Speed Limit and Thermodynamic Uncertainty Relation
• Deep Knockoffs
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Convergence of latent mixing measures in finite and infinite mixture models
• Graph Information Bottleneck
• On integral probability metrics, ϕ-divergences and binary classification
• How to Train Your Energy-Based Models
• Learning Robust Representations via Multi-View Information Bottleneck
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions
• Formal Limitations on the Measurement of Mutual Information
• Ensemble estimation of multivariate f-divergence
• Estimating time-dependent entropy production from non-equilibrium trajectories
• Adversarial-Residual-Coarse-Graining: Applying machine learning theory to systematic molecular coarse-graining
• Universal and Composite Hypothesis Testing via Mismatched Divergence
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
• Generative Adversarial Nets from a Density Ratio Estimation Perspective
• Variational Information Maximization for Feature Selection
• Robust Validation: Confident Predictions Even When Distributions Shift
• Contrastive Variational Autoencoder Enhances Salient Features
• Unsupervised Belief Representation Learning with Information-Theoretic Variational Graph Auto-Encoders
• Understanding the Limitations of Variational Mutual Information Estimators
• On distributionally robust extreme value analysis
• A Theory of Usable Information Under Computational Constraints
• Multivariate f-Divergence Estimation With Confidence
• Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning
• Semi-Supervised Learning of Class Balance under Class-Prior Change by Distribution Matching
• Approximate Inference with Amortised MCMC
• Causal Imitation Learning with Unobserved Confounders
• A Divergence Minimization Perspective on Imitation Learning Methods
• FCC-GAN: A Fully Connected and Convolutional Net Architecture for GANs
• Robust Estimation and Generative Adversarial Nets
• Ensemble Estimation of Information Divergence
• Relevance Factor VAE: Learning and Identifying Disentangled Factors
• Neural Approximate Sufficient Statistics for Implicit Models
• Is novelty predictable?
• Learning Optimal Representations with the Decodable Information Bottleneck
• Wasserstein Dependency Measure for Representation Learning
• Conditional Mutual Information Neural Estimator
• Experience Replay with Likelihood-free Importance Weights
• Conditional Generative Neural System for Probabilistic Trajectory Prediction
• On Data-Processing and Majorization Inequalities for $f$-Divergences with Applications
• Minimax Rate-Optimal Estimation of Divergences between Discrete Distributions
• Kernels on Sample Sets via Nonparametric Divergence Estimates
• Quantum Neural Estimation of Entropies
• Tighter Variational Representations of f-Divergences via Restriction to Probability Measures
• On the Effectiveness of Least Squares Generative Adversarial Networks
• Demystifying Fixed k-Nearest Neighbor Information Estimators
• A case for new neural network smoothness constraints
• Inexact iterative numerical linear algebra for neural network-based spectral estimation and rare-event prediction
• Direct estimation of density functionals using a polynomial basis
• Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges
• Variational Bayesian Optimal Experimental Design with Normalizing Flows
• Learning Uncertainties the Frequentist Way: Calibration and Correlation in High Energy Physics
• Higher-order Efficiency Bound and Its Application to Nonlinear Nano-thermoelectrics
• The Inductive Bias of Restricted f-GANs
• Learning-Aided Physical Layer Attacks Against Multicarrier Communications in IoT
• Training Deep Energy-Based Models with f-Divergence Minimization
• Influence Functions for Machine Learning: Nonparametric Estimators for Entropies, Divergences and Mutual Informations
• Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification
• Solving Inverse Stochastic Problems from Discrete Particle Observations Using the Fokker-Planck Equation and Physics-informed Neural Networks
• Doubly Contrastive Deep Clustering
• Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
• Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning
• Martingale Methods for Sequential Estimation of Convex Functionals and Divergences
• Variational f-divergence Minimization
• Exponential Concentration of a Density Functional Estimator
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
• Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
• Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling
• Direct Density-Derivative Estimation and Its Application in KL-Divergence Approximation
• Offline Reinforcement Learning with Soft Behavior Regularization
• Multi-label Contrastive Predictive Coding
• Preserving physically important variables in optimal event selections: A case study in Higgs physics
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Bayesian Experimental Design for Implicit Models by Mutual Information Neural Estimation
• Constructive Setting of the Density Ratio Estimation Problem and its Rigorous Solution
• Disentangled Representation Learning with Wasserstein Total Correlation
• Negative Data Augmentation
• Geometric Entropic Exploration
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• Information Losses in Neural Classifiers from Sampling
• To Split or Not to Split: The Impact of Disparate Treatment in Classification
• Data-Efficient Mutual Information Neural Estimator
• Asymptotic Guarantees for Generative Modeling Based on the Smooth Wasserstein Distance
• A precise bare simulation approach to the minimization of some distances. Foundations
• Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods
• Estimation of KL Divergence: Optimal Minimax Rate
• Meta learning of bounds on the Bayes classifier error
• Capacity Bounded Differential Privacy
• $(f,\Gamma )$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics
• Obfuscation via Information Density Estimation
• Variational Leakage: The Role of Information Complexity in Privacy Leakage
• Neural Methods for Point-wise Dependency Estimation
• Two-sample inference for high-dimensional Markov networks
• Learning and Strongly Truthful Multi-Task Peer Prediction: A Variational Approach
• Expected Information Maximization: Using the I-Projection for Mixture Density Estimation
• Continuous Doubly Constrained Batch Reinforcement Learning
• Empirically Estimable Classification Bounds Based on a New Divergence Measure
• Properties of f-divergences and f-GAN training
• DEMI: Discriminative Estimator of Mutual Information
• Discriminative Mutual Information Estimators for Channel Capacity Learning
• On Characterizing GAN Convergence Through Proximal Duality Gap
• Variational Sequential Optimal Experimental Design using Reinforcement Learning
• Equivalence between Time Series Predictability and Bayes Error Rate
• PAC Prediction Sets Under Covariate Shift
• Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation
• Telescoping Density-Ratio Estimation
• Novel Change of Measure Inequalities with Applications to PAC-Bayesian Bounds and Monte Carlo Estimation
• Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference
• Quantization via Empirical Divergence Maximization
• Practical and Consistent Estimation of f-Divergences
• Truthful Data Acquisition via Peer Prediction
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion
• Distributed generation of privacy preserving data with user customization
• InfoFair: Information-Theoretic Intersectional Fairness
• Distance Measure Machines
• Nearest neighbor density functional estimation from inverse Laplace transform
• On the Estimation of Information Measures of Continuous Distributions
• Attaining entropy production and dissipation maps from Brownian movies via neural networks
• Deep Dimension Reduction for Supervised Representation Learning
• Self-supervised Representation Learning with Relative Predictive Coding
• Predictive Coding for Locally-Linear Control
• Generalized Energy Based Models
• Minimax Optimal Estimation of KL Divergence for Continuous Distributions
• Statistical Inference after Kernel Ridge Regression Imputation under item nonresponse
• Improving Model Robustness with Latent Distribution Locally and Globally
• A Scalable Gradient-Free Method for Bayesian Experimental Design with Implicit Models
• Learning Disentangled Representations with Latent Variation Predictability
• Capacity-Approaching Autoencoders for Communications
• Gradient-based Bayesian Experimental Design for Implicit Models using Mutual Information Lower Bounds
• Adversarial Disentanglement with Grouped Observations
• Power System Event Identification based on Deep Neural Network with Information Loading
• KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support
• Multimodal Image-to-Image Translation via Mutual Information Estimation and Maximization
• Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity
• Variational Representations and Neural Network Estimation of Rényi Divergences
• Doubly Semi-Implicit Variational Inference
• Quantifying the Task-Specific Information in Text-Based Classifications
• Generating private data with user customization
• General Probabilistic Surface Optimization and Log Density Estimation
• C-MI-GAN : Estimation of Conditional Mutual Information using MinMax formulation
• Adversarial Networks and Autoencoders: The Primal-Dual Relationship and Generalization Bounds
• A Novel Framework for Selection of GANs for an Application
• An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards
• Weighted Classification Cascades for Optimizing Discovery Significance in the HiggsML Challenge
• Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey
• TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learning
• A Distributionally Robust Optimization Method for Adversarial Multiple Kernel Learning
• A Deep Generative Approach to Conditional Sampling
• Dependence Induced Representations
• Exponentially Consistent Kernel Two-Sample Tests
• Fairness-Aware Learning with Restriction of Universal Dependency using f-Divergences
• On the Theoretical Equivalence of Several Trade-Off Curves Assessing Statistical Proximity
• Revisiting Factorizing Aggregated Posterior in Learning Disentangled Representations
• Dependence Maximizing Temporal Alignment via Squared-Loss Mutual Information
• Blocked and Hierarchical Disentangled Representation From Information Theory Perspective
• Off-policy Learning for Multiple Loggers
• f-divergence estimation and two-sample homogeneity test under semiparametric density-ratio models
• Variational Representations of Annealing Paths: Bregman Information under Monotonic Embedding
• Asymptotically Optimal One- and Two-Sample Testing with Kernels
• Maximum Entropy classification for record linkage
• Universal Outlying sequence detection For Continuous Observations
• Divergence Measures Estimation and Its Asymptotic Normality Theory Using Wavelets Empirical Processes
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Measure-conditional Discriminator with Stationary Optimum for GANs and Statistical Distance Surrogates
• LeanML: A Design Pattern To Slash Avoidable Wastes in Machine Learning Projects
• Learning Augmentation Distributions using Transformed Risk Minimization
• Variational Mutual Information Maximization Framework for VAE Latent Codes with Continuous and Discrete Priors
• Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space
• Informative GANs via Structured Regularization of Optimal Transport
• Distributionally Robust Variance Minimization: Tight Variance Bounds over $f$-Divergence Neighborhoods
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories
• Information Theoretic Structured Generative Modeling
• Neural Estimation of Statistical Divergences
• Kullback-Leibler Divergence-Based Out-of-Distribution Detection with Flow-Based Generative Models
• Analysis of Discriminator in RKHS Function Space for Kullback-Leibler Divergence Estimation
• SepVAE: a contrastive VAE to separate pathological patterns from healthy ones
• Performance Guarantees for Quantum Neural Estimation of Entropies
• Bridging the Gap Between $f$-GANs and Wasserstein GANs
• A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification
• $\rho$-VAE: Autoregressive parametrization of the VAE encoder
• On Study of Mutual Information and its Estimation Methods
• Leveraging Hidden Structure in Self-Supervised Learning
• Improving Multimodal fusion via Mutual Dependency Maximisation
• Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
• Divergence measures estimation and its asymptotic normality theory in the discrete case
• Conditional Density Estimation with Dimensionality Reduction via Squared-Loss Conditional Entropy Minimization
• Learning Disentangled Latent Factors from Paired Data in Cross-Modal Retrieval: An Implicit Identifiable VAE Approach
• Information-Corrected Estimation: A Generalization Error Reducing Parameter Estimation Method
• Uplift Modeling from Separate Labels
• Robust W-GAN-Based Estimation Under Wasserstein Contamination
• Continual Density Ratio Estimation in an Online Setting
• Dependence Measures Bounding the Exploration Bias for General Measurements
• Multicalibrated Partitions for Importance Weights
• Non-Asymptotic Performance Guarantees for Neural Estimation of $\mathsf{f}$-Divergences
• Consistency Bands for divergences measures
• Wald-Kernel: Learning to Aggregate Information for Sequential Inference
• Guess First to Enable Better Compression and Adversarial Robustness
• co-BPM: a Bayesian Model for Divergence Estimation
• A Quantitative Evaluation Framework for Missing Value Imputation Algorithms
• Imitation with Neural Density Models