Amplifying Inter-message Distance: On Information Divergence Measures in Big Data

Abstract

Message identification (M-I) divergence is an important measure of the information distance between probability distributions, similar to Kullback-Leibler (K-L) and Renyi divergence. In fact, M-I divergence with a variable parameter can make an effect on characterization of distinction between two distributions. Furthermore, by choosing an appropriate parameter of M-I divergence, it is possible to amplify the information distance between adjacent distributions while maintaining enough gap between two nonadjacent ones. Therefore, M-I divergence can play a vital role in distinguishing distributions more clearly. In this paper, we first define a parametric M-I divergence in the view of information theory and then present its major properties. In addition, we design a M-I divergence estimation algorithm by means of the ensemble estimator of the proposed weight kernel estimators, which can improve the convergence of mean squared error from $O({\Gamma }^{-j/d})$ to $O({\Gamma }^{-1})$ $(j\in (0,d])$. We also discuss the decision with M-I divergence for clustering or classification, and investigate its performance in a statistical sequence model of big data for the outlier detection problem.

arXiv comment30 pages, 4 figures

References in corpus (4)

• Rényi Divergence and Kullback-Leibler Divergence
• On Pinsker's Type Inequalities and Csiszar's f-divergences. Part I: Second and Fourth-Order Inequalities
• Focusing on a Probability Element: Parameter Selection of Message Importance Measure in Big Data
• Non-parametric Message Important Measure: Storage Code Design and Transmission Planning for Big Data

Cited by in corpus (6)

• Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges
• From MIM-Based GAN to Anomaly Detection:Event Probability Influence on Generative Adversarial Networks
• How Many Samples Required in Big Data Collection: A Differential Message Importance Measure
• MIM-Based GAN: Information Metric to Amplify Small Probability Events Importance in Generative Adversarial Networks
• A Switch to the Concern of User: Importance Coefficient in Utility Distribution and Message Importance Measure
• Differential Message Importance Measure: A New Approach to the Required Sampling Number in Big Data Structure Characterization