Importance of Small Probability Events in Big Data: Information Measures, Applications, and Challenges

Abstract

In many applications (e.g., anomaly detection and security systems) of smart cities, rare events dominate the importance of the total information of big data collected by Internet of Things (IoTs). That is, it is pretty crucial to explore the valuable information associated with the rare events involved in minority subsets of the voluminous amounts of data. To do so, how to effectively measure the information with importance of the small probability events from the perspective of information theory is a fundamental question. This paper first makes a survey of some theories and models with respect to importance measures and investigates the relationship between subjective or semantic importance and rare events in big data. Moreover, some applications for message processing and data analysis are discussed in the viewpoint of information measures. In addition, based on rare events detection, some open challenges related to information measures, such as smart cities, autonomous driving, and anomaly detection in IoTs, are introduced which can be considered as future research directions.

References in corpus (14)

• The large deviation approach to statistical mechanics
• Rényi Divergence and Kullback-Leibler Divergence
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Wireless Communications in the Era of Big Data
• Universal Estimation of Directed Information
• Does Dirichlet Prior Smoothing Solve the Shannon Entropy Estimation Problem?
• Maximum Likelihood Estimation of Functionals of Discrete Distributions
• Ensemble estimation of multivariate f-divergence
• Preconditioned Data Sparsification for Big Data with Applications to PCA and K-means
• Multi-criteria Similarity-based Anomaly Detection using Pareto Depth Analysis
• Focusing on a Probability Element: Parameter Selection of Message Importance Measure in Big Data
• Amplifying Inter-message Distance: On Information Divergence Measures in Big Data
• Matching Users' Preference Under Target Revenue Constraints in Optimal Data Recommendation Systems
• The Linear Information Coupling Problems

Cited by in corpus (2)

• An Importance Aware Weighted Coding Theorem Using Message Importance Measure
• Storage Space Allocation Strategy for Digital Data with Message Importance