A precise bare simulation approach to the minimization of some distances. Foundations
arXiv:2107.01693 · doi:10.1109/TIT.2022.3215496
Abstract
In information theory -- as well as in the adjacent fields of statistics, machine learning, artificial intelligence, signal processing and pattern recognition -- many flexibilizations of the omnipresent Kullback-Leibler information distance (relative entropy) and of the closely related Shannon entropy have become frequently used tools. To tackle corresponding constrained minimization (respectively maximization) problems by a newly developed dimension-free bare (pure) simulation method, is the main goal of this paper. Almost no assumptions (like convexity) on the set of constraints are needed, within our discrete setup of arbitrary dimension, and our method is precise (i.e., converges in the limit). As a side effect, we also derive an innovative way of constructing new useful distances/divergences. To illustrate the core of our approach, we present numerous solved cases. The potential for widespread applicability is indicated, too; in particular, we deliver many recent references for uses of the involved distances/divergences and entropies in various different research fields (which may also serve as an interdisciplinary interface).
arXiv commentv3: considerably shortened and restructured version of v1/v2; 64 pages + 7 pages supplement. This work is accepted by the journal "IEEE Transactions on Information Theory", and is available in early-access form at https://ieeexplore.ieee.org/document/9925151
References in corpus (41)
- Rényi Divergence and Kullback-Leibler Divergence
- Quantum Random Number Generators
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Deep neural networks for the evaluation and design of photonic devices
- Operationally Invariant Information in Quantum Measurements
- A Reconfigurable Nanophotonics Platform for Sub-Millisecond, Deep Brain Neural Stimulation
- Unsupervised Image Regression for Heterogeneous Change Detection
- Massively parallel ultrafast random bit generation with a chip-scale laser
- Guessing under source uncertainty
- An information-theoretic, all-scales approach to comparing networks
- Topology comparison of Twitter diffusion networks effectively reveals misleading information
- Pre-print: Radio Identity Verification-based IoT Security Using RF-DNA Fingerprints and SVM
- Encoding Tasks and Rényi Entropy
- On Bregman Distances and Divergences of Probability Measures
- Uncertainty-Aware Principal Component Analysis
- Minimization of divergences on sets of signed measures
- Certified Quantum Random Numbers from Untrusted Light
- Quadratic distances on probabilities: A unified foundation
- The entropic basis of collective behaviour
- Several Applications of Divergence Criteria in Continuous Families
- On distributionally robust extreme value analysis
- Minimization of entropy functionals
- Estimating the spectrum in computed tomography via Kullback-Leibler divergence constrained optimization
- Building and using semiparametric tolerance regions for parametric multinomial models
- A Generalized Asymmetric Dual-front Model for Active Contours and Image Segmentation
- Maximum-Entropy-Rate Selection of Features for Classifying Changes in Knee and Ankle Dynamics During Running
- On Data-Processing and Majorization Inequalities for -Divergences with Applications
- Collective Awareness for Abnormality Detection in Connected Autonomous Vehicles
- A Scale-invariant Generalization of the Rényi Entropy, Associated Divergences and their Optimizations under Tsallis' Nonextensive Framework
- Thermodynamics of Random Number Generation
- Tight Bounds for Symmetric Divergence Measures and a New Inequality Relating -Divergences
- A New Family of Divergences Originating from Model Adequacy Tests and Application to Robust Statistical Inference
- Uncovering the hierarchical structure of the international FOREX market by using similarity metric between the fluctuation distributions of currencies
- Multiple Private Key Generation for Continuous Memoryless Sources with A Helper
- Mathematical Modelling of Streamwise Velocity Profile in Open Channels Using Tsallis Entropy
- Optimality of Rate Balancing in Wireless Sensor Networks
- Minimax Rényi Redundancy
- Renyi entropy as a measure of cosmic homogeneity
- Some distance bounds of branching processes and their diffusion limits
- A constructive method to minimize couple matchings
- Optimal transport with some directed distances