Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space
arXiv:2109.14688
Abstract
Estimating Kullback Leibler (KL) divergence from samples of two distributions is essential in many machine learning problems. Variational methods using neural network discriminator have been proposed to achieve this task in a scalable manner. However, we noted that most of these methods using neural network discriminators suffer from high fluctuations (variance) in estimates and instability in training. In this paper, we look at this issue from statistical learning theory and function space complexity perspective to understand why this happens and how to solve it. We argue that the cause of these pathologies is lack of control over the complexity of the neural network discriminator function and could be mitigated by controlling it. To achieve this objective, we 1) present a novel construction of the discriminator in the Reproducing Kernel Hilbert Space (RKHS), 2) theoretically relate the error probability bound of the KL estimates to the complexity of the discriminator in the RKHS space, 3) present a scalable way to control the complexity (RKHS norm) of the discriminator for a reliable estimation of KL divergence, and 4) prove the consistency of the proposed estimator. In three different applications of KL divergence : estimation of KL, estimation of mutual information and Variational Bayes, we show that by controlling the complexity as developed in the theory, we are able to reduce the variance of KL estimates and stabilize the training
arXiv comment27 pages, 3 figures. arXiv admin note: text overlap with arXiv:2002.11187
References in corpus (7)
- Understanding deep learning requires rethinking generalization
- Which Training Methods for GANs do actually Converge?
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
- Amortised MAP Inference for Image Super-resolution
- On Variational Bounds of Mutual Information
- Maximum Margin Multiclass Nearest Neighbors