Notes on Kullback-Leibler Divergence and Likelihood

Abstract

The Kullback-Leibler (KL) divergence is a fundamental equation of information theory that quantifies the proximity of two probability distributions. Although difficult to understand by examining the equation, an intuition and understanding of the KL divergence arises from its intimate relationship with likelihood theory. We discuss how KL divergence arises from likelihood theory in an attempt to provide some intuition and reserve a rigorous (but rather simple) derivation for the appendix. Finally, we comment on recent applications of KL divergence in the neural coding literature and highlight its natural application.

Cited by in corpus (11)

• Optimization models of natural communication
• Unsupervised Neural Domain Adaptation for Document Image Binarization
• Latent Denoising Diffusion GAN: Faster sampling, Higher image quality
• COVID-19: Detecting Depression Signals during Stay-At-Home Period
• Simulating Personal Food Consumption Patterns using a Modified Markov Chain
• Exploring Public Attention in the Circular Economy through Topic Modelling with Twin Hyperparameter Optimisation
• FFPDG: Fast, Fair and Private Data Generation
• Measure of gap and inequalities in basic education students proficiencies
• Biased Models Have Biased Explanations
• An Introduction to Variational Inference
• Wasserstein Distance Guided Cross-Domain Learning