Estimating Mutual Information

Abstract

We present two classes of improved estimators for mutual information $M(X,Y)$, from samples of random points distributed according to some joint probability density $\mu (x,y)$. In contrast to conventional estimators based on binnings, they are based on entropy estimates from $k$-nearest neighbour distances. This means that they are data efficient (with $k=1$ we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to non-uniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of $k/N$ for $N$ points. Numerically, we find that both families become {\it exact} for independent distributions, i.e. the estimator $\hat{M}(X,Y)$ vanishes (up to statistical fluctuations) if $\mu (x,y)=\mu (x)\mu (y)$. This holds for all tested marginal distributions and for all dimensions of $x$ and $y$. In addition, we give estimators for redundancies between more than 2 random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation.

16 pages, including 18 figures

Cited by in corpus (489)

• A Review of Feature Selection Methods Based on Mutual Information
• Granger causality and transfer entropy are equivalent for Gaussian variables
• Nonlinear multivariate analysis of Neurophysiological Signals
• Detecting causal associations in large nonlinear time series datasets
• Opening the Black Box of Deep Neural Networks via Information
• Equitability, mutual information, and the maximal information coefficient
• Complex networks in climate dynamics - Comparing linear and nonlinear network construction methods
• JIDT: An information-theoretic toolkit for studying the dynamics of complex systems
• Distinguishing cause from effect using observational data: methods and benchmarks
• A Novel Combined Data-Driven Approach for Electricity Theft Detection
• HERMES: towards an integrated toolbox to characterize functional and effective brain connectivity
• On Mutual Information Maximization for Representation Learning
• Predicting the thermodynamic stability of perovskite oxides using machine learning models
• A class of Rényi information estimators for multidimensional densities
• Causal Network Inference by Optimal Causation Entropy
• Mutual information for the selection of relevant variables in spectrometric nonlinear modelling
• Non-uniform state space reconstruction and coupling detection
• Least Dependent Component Analysis Based on Mutual Information
• Causation Entropy Identifies Indirect Influences, Dominance of Neighbors and Anticipatory Couplings
• Quantum Machine Learning for Chemistry and Physics
• Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data
• Quantifying Causal Coupling Strength: A Lag-specific Measure For Multivariate Time Series Related To Transfer Entropy
• Direct coupling information measure from non-uniform embedding
• Neural Network Renormalization Group
• Efficient transfer entropy analysis of non-stationary neural time series
• Transfer Entropy as a Log-likelihood Ratio
• Motor control by precisely timed spike patterns
• Entropy and mutual information in models of deep neural networks
• Bits from Biology for Computational Intelligence
• Granger Causality in Multi-variate Time Series using a Time Ordered Restricted Vector Autoregressive Model
• The relation between Granger causality and directed information theory: a review
• High-$T_c$ superconductivity by mobilizing local spin singlets and possible route to higher $T_c$ in pressurized La${}_3$Ni${}_2$O${}_7$
• Unified functional network and nonlinear time series analysis for complex systems science: The pyunicorn package
• Resampling methods for parameter-free and robust feature selection with mutual information
• Large-scale directed network inference with multivariate transfer entropy and hierarchical statistical testing
• Measuring Integrated Information: Comparison of Candidate Measures in Theory and Simulation
• On directed information theory and Granger causality graphs
• Compressing Neural Networks using the Variational Information Bottleneck
• IDTxl: The Information Dynamics Toolkit xl: a Python package for the efficient analysis of multivariate information dynamics in networks
• MODNet -- accurate and interpretable property predictions for limited materials datasets by feature selection and joint-learning
• Inferring deterministic causal relations
• Estimating Mutual Information for Discrete-Continuous Mixtures
• Persistent Homology of Complex Networks for Dynamic State Detection
• Active learning for data streams: a survey
• Hierarchical Clustering Based on Mutual Information
• Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines
• Nonlinear optical encoding enabled by recurrent linear scattering
• Information-theoretic measurements of coupling between structure and dynamics in glass-formers
• Thinking outside the ROCs: Designing Decorrelated Taggers (DDT) for jet substructure
• Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs
• A Mutual Information Approach to Calculating Nonlinearity
• Assessing coupling dynamics from an ensemble of time series
• Machine Learning-based Urban Canyon Path Loss Prediction using 28 GHz Manhattan Measurements
• Tightening Mutual Information Based Bounds on Generalization Error
• Independent components in spectroscopic analysis of complex mixtures
• Efficient Estimation of Mutual Information for Strongly Dependent Variables
• Partial Transfer Entropy on Rank Vectors
• Large-scale nonlinear Granger causality: A data-driven, multivariate approach to recovering directed networks from short time-series data
• A Topic-Agnostic Approach for Identifying Fake News Pages
• Selecting embedding delays: An overview of embedding techniques and a new method using persistent homology
• Non-linear dependence and teleconnections in climate data: sources, relevance, nonstationarity
• Equitability Analysis of the Maximal Information Coefficient, with Comparisons
• Deep Dictionary Learning: A PARametric NETwork Approach
• Quantifying information transfer and mediation along causal pathways in complex systems
• Separation of pulsar signals from noise with supervised machine learning algorithms
• Measuring dependence powerfully and equitably
• Formal Limitations on the Measurement of Mutual Information
• Geometric correlations mitigate the extreme vulnerability of multiplex networks against targeted attacks
• An evaluation of intrusive instrumental intelligibility metrics
• A non-parametric k-nearest neighbour entropy estimator
• Non-parametric Ensemble Empirical Mode Decomposition for extracting weak features to identify bearing defects
• Excessive Invariance Causes Adversarial Vulnerability
• Evaluation of Mutual Information Estimators for Time Series
• Mutual Information as a Tool for Identifying Phase Transitions in Dynamical Complex Systems With Limited Data
• Breakdown of local information processing may underlie isoflurane anesthesia effects
• How Entropic Regression Beats the Outliers Problem in Nonlinear System Identification
• On Information Plane Analyses of Neural Network Classifiers -- A Review
• Comparison of Methods for the Assessment of Nonlinearity in Short-Term Heart Rate Variability under different Physiopathological States
• Information-theoretic analysis of multivariate single - cell signaling responses using SLEMI
• Inference of Causal Information Flow in Collective Animal Behavior
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• End-to-end correlated subgap states in hybrid nanowires
• An information theoretic approach to statistical dependence: copula information
• Detecting synchronization clusters in multivariate time series via coarse-graining of Markov chains
• Monte Carlo Algorithm for Least Dependent Non-Negative Mixture Decomposition
• Correlation-based feature selection to identify functional dynamics in proteins
• Bayesian global analysis of neutrino oscillation data
• On the Automatic Parameter Selection for Permutation Entropy
• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• Fast Selection of Spectral Variables with B-Spline Compression
• Estimation of time-delayed mutual information and bias for irregularly and sparsely sampled time-series
• An instrumental intelligibility metric based on information theory
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• Holographic entanglement of purification for thermofield double states and thermal quench
• Estimation of mutual information for real-valued data with error bars and controlled bias
• Identifying Coupling Structure in Complex Systems through the Optimal Causation Entropy Principle
• Editorial Comment on the Special Issue of "Information in Dynamical Systems and Complex Systems"
• Determining the Twist Angle of Bilayer Graphene by Machine Learning Analysis of its Raman Spectrum
• Optimal model-free prediction from multivariate time series
• High-Order Conditional Mutual Information Maximization for dealing with High-Order Dependencies in Feature Selection
• Using mutual information to measure order in model glass-formers
• Advances in Feature Selection with Mutual Information
• Mutual information rate and bounds for it
• Solar cycle dependence of spatial correlation in the solar wind
• Multiscale Wavelet Transfer Entropy with Application to Corticomuscular Coupling Analysis
• Information engine in a nonequilibrium bath
• Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information
• Lower Bounds on Mutual Information
• Dynamic sampling and information encoding in biochemical networks
• Robust period estimation using mutual information for multi-band light curves in the synoptic survey era
• Estimating information in time-varying signals
• Successful network inference from time-series data using Mutual Information Rate
• A new method for choosing parameters in delay reconstruction-based forecast strategies
• BET on Independence
• Information estimation using nonparametric copulas
• Early warning signals for critical transitions in complex systems
• A data-driven functional projection approach for the selection of feature ranges in spectra with ICA or cluster analysis
• Likelihood Non-Gaussianity in Large-Scale Structure Analyses
• Linear and nonlinear market correlations: characterizing financial crises and portfolio optimization
• Causation entropy from symbolic representations of dynamical systems
• A Theory of Usable Information Under Computational Constraints
• Functional Connectome of the Human Brain with Total Correlation
• Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet
• Measures of Causality in Complex Datasets with application to financial data
• Information theory for non-stationary processes with stationary increments
• Geometric k-nearest neighbor estimation of entropy and mutual information
• Nonlinear dependencies on Brazilian equity network from mutual information minimum spanning trees
• Extracting Sentence Embeddings from Pretrained Transformer Models
• Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
• The causal inference of cortical neural networks during music improvisations
• Neural Canonical Transformation with Symplectic Flows
• Relating Granger causality to directed information theory for networks of stochastic processes
• Detecting causality in multivariate time series via non-uniform embedding
• Analysis of k-Nearest Neighbor Distances with Application to Entropy Estimation
• Mutual Information in Frequency and its Application to Measure Cross-Frequency Coupling in Epilepsy
• Transfer Entropy on Rank Vectors
• Scaling of Information in Turbulence
• Causality indices for bivariate time series data: a comparative review of performance
• ROSpace: Intrusion Detection Dataset for a ROS2-Based Cyber-Physical System
• Uncertainty Decomposition in Bayesian Neural Networks with Latent Variables
• Do planets remember how they formed?
• Birds of a Feather Flock Together: Satirical News Detection via Language Model Differentiation
• A robust estimator of mutual information for deep learning interpretability
• Accurate estimation of the normalized mutual information of multidimensional data
• Apparent remote synchronization of amplitudes: a demodulation and interference effect
• Causality analysis of large-scale structures in the flow around a wall-mounted square cylinder
• Socioeconomic biases in urban mixing patterns of US metropolitan areas
• Privacy-Cost Management in Smart Meters Using Deep Reinforcement Learning
• Computing spatially resolved rotational hydration entropies from atomistic simulations
• Mutual information between reflected and transmitted speckle images
• Mutual Information Scaling for Tensor Network Machine Learning
• Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples
• Information Flow in Computational Systems
• Calibrating cosmological simulations with implicit likelihood inference using galaxy growth observables
• CCMI : Classifier based Conditional Mutual Information Estimation
• Fuzzy Jaccard Index: A robust comparison of ordered lists
• Band selection for oxygenation estimation with multispectral/hyperspectral imaging
• Estimating Information Flow in Deep Neural Networks
• Path Weight Sampling: Exact Monte Carlo Computation of the Mutual Information between Stochastic Trajectories
• Estimating Conditional Transfer Entropy in Time Series using Mutual Information and Non-linear Prediction
• Wasserstein Dependency Measure for Representation Learning
• Information Based Centralization of Locomotion in Animals and Robots
• Circle fit optimization for resonator quality factor measurements: point redistribution for maximal accuracy
• Symmetries and phase diagrams with real-space mutual information neural estimation
• Exploratory Data Analysis for Airline Disruption Management
• Data Compression and Inference in Cosmology with Self-Supervised Machine Learning
• Conditional Mutual Information Neural Estimator
• Sufficiency of a Gaussian power spectrum likelihood for accurate cosmology from upcoming weak lensing surveys
• Transfer Entropy Rate Through Lempel-Ziv Complexity
• Information Theory to probe Intrapartum Fetal Heart Rate Dynamics
• A Graph Algorithmic Approach to Separate Direct from Indirect Neural Interactions
• End-to-End Intelligent Framework for Rockfall Detection
• Mutual Information for Explainable Deep Learning of Multiscale Systems
• Mutual information, quantum phase transition, and phase coherence in Kondo systems
• Shortcomings of transfer entropy and partial transfer entropy: Extending them to escape the curse of dimensionality
• Causality and Bayesian network PDEs for multiscale representations of porous media
• Demystifying Fixed k-Nearest Neighbor Information Estimators
• Separating a Real-Life Nonlinear Image Mixture
• Deep learning denoising by dimension reduction: Application to the ORION-B line cubes
• Coronal Temperature Maps from Solar EUV images: a Blind Source Separation Approach
• k-core structure of real multiplex networks
• Interpretable Approximation of High-Dimensional Data
• Shape or Texture: Understanding Discriminative Features in CNNs
• HetEmotionNet: Two-Stream Heterogeneous Graph Recurrent Neural Network for Multi-modal Emotion Recognition
• Automatic feature selection and weighting in molecular systems using Differentiable Information Imbalance
• Functional Connectivity via Total Correlation: Analytical results in Visual Areas
• A Causality-Based Learning Approach for Discovering the Underlying Dynamics of Complex Systems from Partial Observations with Stochastic Parameterization
• Estimating Transfer Entropy via Copula Entropy
• Simple and Efficient Partial Graph Adversarial Attack: A New Perspective
• Variable Selection with Copula Entropy
• On quantitative aspects of model interpretability
• Maximizing Information Gain for the Characterization of Biomolecular Circuits
• Information Theoretic Structure Learning with Confidence
• Clickstream analysis for crowd-based object segmentation with confidence
• Discovering Nonlinear Relations with Minimum Predictive Information Regularization
• A layered multiple importance sampling scheme for focused optimal Bayesian experimental design
• A Machine Learning Approach to Predict Missing Flux Densities in Multi-band Galaxy Surveys
• Variational inference of fractional Brownian motion with linear computational complexity
• Encoding protein dynamic information in graph representation for functional residue identification
• Entropic Causal Inference for Neurological Applications
• Estimating differential entropy using recursive copula splitting
• Quantifying the predictability of visual scanpaths using active information storage
• Frustratingly Easy Transferability Estimation
• Learnability for the Information Bottleneck
• Causal Discovery with General Non-Linear Relationships Using Non-Linear ICA
• Understanding Compressive Adversarial Privacy
• Probing high order dependencies with information theory
• GroupLiNGAM: Linear non-Gaussian acyclic models for sets of variables
• Generating functionals for computational intelligence: the Fisher information as an objective function for self-limiting Hebbian learning rules
• Detecting continuous variable entanglement in phase space with the $Q$-distribution
• Inference of forex and stock-index financial networks based on the normalised mutual information rate
• Generating Redundant Features with Unsupervised Multi-Tree Genetic Programming
• Deep or Not Deep: Supervised Learning Approaches to Modeling the Pedestal Density
• Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration
• Entropy estimators for Markovian sequences: A comparative analysis
• Neural Entropic Estimation: A faster path to mutual information estimation
• Information-Theoretic Bounds and Approximations in Neural Population Coding
• AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data -- Accepted Version
• Demystifying Deep Learning in Predictive Spatio-Temporal Analytics: An Information-Theoretic Framework
• Layer-wise Characterization of Latent Information Leakage in Federated Learning
• Shared Information in Stationary States at Criticality
• PDE-Driven Spatiotemporal Disentanglement
• Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification
• An Empirical Study of Leading Measures of Dependence
• Inter-Scale Information Flow as a Surrogate for Downward Causation That Maintains Spiral Waves
• ITENE: Intrinsic Transfer Entropy Neural Estimator
• Discovering Association with Copula Entropy
• Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models
• Kendall transformation: a robust representation of continuous data for information theory
• Causal network discovery by iterative conditioning: comparison of algorithms
• Generating Neural Networks with Neural Networks
• Theoretical Foundations of Equitability and the Maximal Information Coefficient
• Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
• A Framework for Causal Discovery in non-intervenable systems
• Simple stopping criteria for information theoretic feature selection
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
• varrank: an R package for variable ranking based on mutual information with applications to observed systemic datasets
• Fast Fair Regression via Efficient Approximations of Mutual Information
• Enabling Collaborative Data Science Development with the Ballet Framework
• Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling
• $k$-NN Estimation of Directed Information
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• PointSmile: Point Self-supervised Learning via Curriculum Mutual Information
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysis
• Probabilistic Data Analysis with Probabilistic Programming
• Causal Influence Detection for Improving Efficiency in Reinforcement Learning
• Pitfalls of Climate Network Construction: A Statistical Perspective
• Sliced Mutual Information: A Scalable Measure of Statistical Dependence
• Quantifying the informativity of emission lines to infer physical conditions in giant molecular clouds. I. Application to model predictions
• Information Arbitrage in Bipartite Heat Engines
• Fairness-Aware Neural Réyni Minimization for Continuous Features
• Dynamical coupling during collective animal motion
• Exploring Privacy and Fairness Risks in Sharing Diffusion Models: An Adversarial Perspective
• Partial Information Decomposition via Deficiency for Multivariate Gaussians
• A Kernel-Based Calculation of Information on a Metric Space
• Mining higher-order triadic interactions
• Predicting high-dimensional heterogeneous time series employing generalized local states
• Usable Information and Evolution of Optimal Representations During Training
• Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration
• Deciphering High-order Structural Correlations within Fluxional Molecules from Classical and Quantum Configurational Entropy
• Multi-modal machine learning analysis of GaSe molecular beam epitaxy growth conditions
• Quantifying Non-Stationarity with Information Theory
• Mutual Information Gradient Estimation for Representation Learning
• Data-Efficient Mutual Information Neural Estimator
• Estimating Mutual Information by Local Gaussian Approximation
• Copula Index for Detecting Dependence and Monotonicity between Stochastic Signals
• Transfer entropy computation using the Perron-Frobenius operator
• On Geometry of Information Flow for Causal Inference
• Brownian dynamics simulations of an idealized chemical reaction network under spatial confinement and crowding conditions
• Renormalized Mutual Information for Artificial Scientific Discovery
• Learned practical guidelines for evaluating Conditional Entropy and Mutual Information in discovering major factors of response-vs-covariate dynamics
• Mutual Information as a Figure of Merit for Optical Fiber Systems
• A Data-Driven Two-Phase Multi-Split Causal Ensemble Model for Time Series
• A pattern recognition approach for distinguishing between prose and poetry
• Work relations with measurement and feedback control on nonuniform temperature systems
• Neural Methods for Point-wise Dependency Estimation
• Measuring Dependence with Matrix-based Entropy Functional
• Superconducting fluctuations and charge-4$e$ plaquette state at strong coupling
• Geometric Estimation of Multivariate Dependency
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• Non-Global Correlations in Collider Physics
• An approximate KLD based experimental design for models with intractable likelihoods
• Clarifying the conceptual dimensions of representation in neuroscience
• Dynamic learning rate using Mutual Information
• Brain Topography Adaptive Network for Satisfaction Modeling in Interactive Information Access System
• Load Modulation for Backscatter Communication: Channel Capacity and Near-Capacity Schemes
• Network inference combining mutual information rate and statistical tests
• Unsupervised Information Obfuscation for Split Inference of Neural Networks
• A Rigorous Information-Theoretic Definition of Redundancy and Relevancy in Feature Selection Based on (Partial) Information Decomposition
• Entropy Estimation via Uniformization
• Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding
• Discriminative Mutual Information Estimators for Channel Capacity Learning
• Estimating Information-Theoretic Quantities
• Balanced Information Storage and Transfer in Modular Spiking Neural Networks
• Keys from the Sky: A First Exploration of Physical-Layer Security Using Satellite Links
• Power- and Spectral Efficient Communication System Design Using 1-Bit Quantization
• Towards Better Long-range Time Series Forecasting using Generative Adversarial Networks
• Dependence Structure Estimation via Copula
• Thermodynamics of Encoding and Encoders
• Information dynamics of price and liquidity around the 2017 Bitcoin markets crash
• Failure Modes of Variational Autoencoders and Their Effects on Downstream Tasks
• Velocity offset between emission and absorption lines might be an effective indicator of dual core system
• Climate entropy production recorded in a deep Antarctic ice core
• Quantifying Emergent Behavior of Autonomous Robots
• Integrated Information Decomposition Unveils Major Structural Traits of $In$ $Silico$ and $In$ $Vitro$ Neuronal Networks
• copent: Estimating Copula Entropy and Transfer Entropy in R
• Pareto-optimal clustering with the primal deterministic information bottleneck
• Essential conditions for evolution of communication within a species
• Recurrence flow measure of nonlinear dependence
• DEMI: Discriminative Estimator of Mutual Information
• Evolvability signatures of generative encodings: beyond standard performance benchmarks
• Nearest neighbor density functional estimation from inverse Laplace transform
• On a comparative study between dependence scales determined by linear and non-linear measures
• Applications of Information Theory to Analysis of Neural Data
• Are Deep Neural Architectures Losing Information? Invertibility Is Indispensable
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching
• Area laws and thermalization from classical entropies in a Bose-Einstein condensate
• Privacy-oriented manipulation of speaker representations
• Adaptive Sampling of 3D Spatial Correlations for Focus+Context Visualization
• Variational Monte Carlo Approach to Partial Differential Equations with Neural Networks
• A Channel Coding Benchmark for Meta-Learning
• Tensor networks and efficient descriptions of classical data
• Information Theory Measures via Multidimensional Gaussianization
• Distribution-free consistent independence tests via center-outward ranks and signs
• Bundled Causal History Interaction
• Quantifying dependencies for sensitivity analysis with multivariate input sample data
• Anomalous Behaviour of Mutual Information in Finite Flocks
• Techniques for noise removal from EEG, EOG and air flow signals in sleep patients
• Asymmetric Graph Error Control with Low Complexity in Causal Bandits
• Comparison of global sensitivity analysis methods for a fire spread model with a segmented characteristic
• Ensemble Estimation of Distributional Functionals via $k$-Nearest Neighbors
• Entropy estimation in a spin-1 Bose-Einstein condensate
• Batch norm with entropic regularization turns deterministic autoencoders into generative models
• Target-Focused Feature Selection Using a Bayesian Approach
• Learning Sparse Privacy-Preserving Representations for Smart Meters Data
• Efficient sampling of non log-concave posterior distributions with mixture of noises
• Can Information Flows Suggest Targets for Interventions in Neural Circuits?
• Application of Mutual Information Methods in Time-Distance Helioseismology
• A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Feature Selection Based on Unique Relevant Information for Health Data
• Random Forests for Adaptive Nearest Neighbor Estimation of Information-Theoretic Quantities
• Capacity-Approaching Autoencoders for Communications
• PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders
• Mutual Information State Intrinsic Control
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• Fundamental limitations to key distillation from Gaussian states with Gaussian operations
• Mutual Information in Molecular and Macromolecular Systems
• SparseEMG: Computational Design of Sparse EMG Layouts for Sensing Gestures
• On the Upper Limit of Separability
• Towards Accurate Forecasting of Renewable Energy : Building Datasets and Benchmarking Machine Learning Models for Solar and Wind Power in France
• Towards a robust approach to infer causality in molecular systems satisfying detailed balance
• Using Linear Dynamical Topic Model for Inferring Temporal Social Correlation in Latent Space
• Necessary and sufficient graphical conditions for optimal adjustment sets in causal graphical models with hidden variables
• Information-Theoretic Representation Learning for Positive-Unlabeled Classification
• Understanding Concept Identification as Consistent Data Clustering Across Multiple Feature Spaces
• Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning
• SentenceMIM: A Latent Variable Language Model
• Associations between finger tapping, gait and fall risk with application to fall risk assessment
• Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
• Bounds on mutual information of mixture data for classification tasks
• Modes of Information Flow
• Information Bottleneck: Exact Analysis of (Quantized) Neural Networks
• Inferring a property of a large system from a small number of samples
• Prediction and Mitigation of Crush Conditions in Emergency Evacuations
• Siamese Learning-based Monarch Butterfly Localization
• Generalization of LiNGAM that allows confounding
• Efficiently Estimating Mutual Information Between Attributes Across Tables
• On a General Theoretical Framework of Reliability
• MIC: Mutual Information based hierarchical Clustering
• Phase space distributions in information theory
• Buses of Cuernavaca - an agent-based model for universal random matrix behavior minimizing mutual information
• Approximate Computational Approaches for Bayesian Sensor Placement in High Dimensions
• The Information Geometry of Unsupervised Reinforcement Learning
• Biased Models Have Biased Explanations
• C-MI-GAN : Estimation of Conditional Mutual Information using MinMax formulation
• Understanding confounding effects in linguistic coordination: an information-theoretic approach
• End-coupled random lasers: a basis for artificial neural networks
• MIM: Mutual Information Machine
• Long-range Prediction of Vital Signs Using Generative Boosting via LSTM Networks
• Emergence of synchrony in an Adaptive Interaction Model
• Adaptive Geo-Topological Independence Criterion
• Interpreting Deep Neural Network-Based Receiver Under Varying Signal-To-Noise Ratios
• Free as in Free Word Order: An Energy Based Model for Word Segmentation and Morphological Tagging in Sanskrit
• Generalized R-squared for Detecting Dependence
• Kernelized Hashcode Representations for Relation Extraction
• Data-Driven Estimation Of Mutual Information Between Dependent Data
• Conditional Mutual Information Estimation for Mixed Discrete and Continuous Variables with Nearest Neighbors
• Directional Localization in Disordered 2D Tight-Binding Systems: Insights from Single Particle Entanglement Measures
• Leveraging WiFi Network Logs to Infer Student Collocation and its Relationship with Academic Performance
• Identifying statistical indicators of temporal asymmetry using a data-driven approach
• Regularized Estimation of Information via High Dimensional Canonical Correlation Analysis
• Inference, Prediction, and Entropy-Rate Estimation of Continuous-time, Discrete-event Processes
• Disentangling Video with Independent Prediction
• Feature Selection for multi-labeled variables via Dependency Maximization
• Measuring Information Leakage in Website Fingerprinting Attacks and Defenses
• Unravelling the Flow of Information in a Nonequilibrium Process in the Presence of Hydrodynamic Interactions
• Predicting MMSE Score from Finger-Tapping Measurement
• Bacterial chemotaxis: information processing, thermodynamics, and behavior
• A concise method for feature selection via normalized frequencies
• Learning the Arrow of Time
• From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation
• Fisher exact scanning for dependency
• What drives bitcoin? An approach from continuous local transfer entropy and deep learning classification models
• fastMI: a fast and consistent copula-based estimator of mutual information
• Homogenizing Entropy Across Different Environmental Conditions: A Universally Applicable Method for Transforming Continuous Variables
• Inferring the finest pattern of mutual independence from data
• Modeling Psychotherapy Dialogues with Kernelized Hashcode Representations: A Nonparametric Information-Theoretic Approach
• Robust Non-linear Wiener-Granger Causality For Large High-dimensional Data
• Consistent Entropy Estimation for Stationary Time Series
• Finding Optimal Diverse Feature Sets with Alternative Feature Selection
• Bounds on the rates of statistical divergences and mutual information via stochastic thermodynamics
• Inverse Learning of Symmetries
• Quantifying Emergence in terms of Persistent Mutual Information
• Short-term Wind Speed Forecasting based on LSSVM Optimized by Elitist QPSO
• Estimation of mutual information via quantum kernel method
• Modeling and Analysis of the Lead-Lag Network of Economic Indicators
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Understanding Neural Networks with Logarithm Determinant Entropy Estimator
• Equitability of Dependence Measure
• Learning Whole-Body Human-Robot Haptic Interaction in Social Contexts
• Information-Theoretic Measures of Influence Based on Content Dynamics
• Mutual Information in Coupled Double Quantum Dots: A Simple Analytic Model for Potential Artificial Consciousness
• Coupling Distant Annotation and Adversarial Training for Cross-Domain Chinese Word Segmentation
• maigesPack: A Computational Environment for Microarray Data Analysis
• Generalized Pearson correlation squares for capturing mixtures of bivariate linear dependences
• Bayesian experimental design for the active nitridation of graphite by atomic nitrogen
• Bayesian information-theoretic calibration of patient-specific radiotherapy sensitivity parameters for informing effective scanning protocols in cancer
• Dynamic Feature Acquisition with Arbitrary Conditional Flows
• Malicious Network Traffic Detection via Deep Learning: An Information Theoretic View
• Non parametric estimation of joint, Renyi-Stallis entropies and mutual information and asymptotic limits
• Global Sensitivity Analysis with Dependence Measures
• High Mutual Information in Representation Learning with Symmetric Variational Inference
• Secret-key generation from wireless channels: Mind the reflections
• Model-Augmented Estimation of Conditional Mutual Information for Feature Selection
• Morphology-Agnostic Visual Robotic Control
• Computing Similarity Queries for Correlated Gaussian Sources
• Parametric inference in the large data limit using maximally informative models
• Achievable Rates of Attack Detection Strategies in Echo-Assisted Communication
• Uncovering networks amongst stocks returns by studying nonlinear interactions in high frequency data of the Indian Stock Market using mutual information
• Plant-wide fault and disturbance screening using combined transfer entropy and eigenvector centrality analysis
• On the Detection of Mutual Influences and Their Consideration in Reinforcement Learning Processes
• Digital System Reconstruction by Pairwise Transfer Entropy
• Probabilistic CCA with Implicit Distributions
• Mutual information in changing environments: non-linear interactions, out-of-equilibrium systems, and continuously-varying diffusivities
• Building Models for Biopathway Dynamics Using Intrinsic Dimensionality Analysis
• Hidden multiscale organization and robustness of real multiplex networks
• Bayesian Regression Markets
• TCMI: a non-parametric mutual-dependence estimator for multivariate continuous distributions
• Effective Connectivity-Based Neural Decoding: A Causal Interaction-Driven Approach
• Entropic Regression DMD (ERDMD) Discovers Informative Sparse and Nonuniformly Time Delayed Models
• Functional Dynamical Structures in Complex Systems: an Information-Theoretic Approach
• Entropy-based Discovery of Summary Causal Graphs in Time Series
• Information Flow in Finite Flocks with Topological Interactions
• Improving Numerical Stability of Normalized Mutual Information Estimator on High Dimensions
• Speckle statistics in adaptive optics images at visible wavelengths
• Diffusion Model-based Parameter Estimation in Dynamic Power Systems
• On the effectiveness of a binless entropy estimator for generalised entropic forms
• Analysis of KNN Information Estimators for Smooth Distributions
• A Novel Receiver Design with Joint Coherent and Non-Coherent Processing
• Adversarial Orthogonal Regression: Two non-Linear Regressions for Causal Inference
• Quantifying Influence and Information Transfer in a Modified Vicsek Model with Non-reciprocal Interactions
• Photonic Learning in Ultrafast Laser-Induced Complexity
• Identification of pressure points in modern power systems using transfer entropy
• On Mutual Information Neural Estimation for Localization
• Data-Driven Reduction of Fault Location Errors in Onshore Wind Farm Collectors
• Probing information theoretic measures of nonlinear ultracold quantum gases using phase-space distributions
• Quantum mutual information, coherence and unified relations of top quarks in QCD processes
• Specific Differential Entropy Rate Estimation for Continuous-Valued Time Series
• Transfer entropy and O-information to detect grokking in tensor network multi-class classification problems
• Reinforced Domain Selection for Continuous Domain Adaptation
• Robust Predictions with Ambiguous Time Delays: A Bootstrap Strategy
• A note on the unbiased estimation of mutual information
• Learning Structures for Deep Neural Networks
• Measuring and Discovering Correlations in Large Data Sets
• Inferring Volatility in the Heston Model and its Relatives -- an Information Theoretical Approach
• A Data-driven Explainable Case-based Reasoning Approach for Financial Risk Detection
• A Bayesian Nonparametric Estimation of Mutual Information
• Learning Opinion Summarizers by Selecting Informative Reviews
• Efficient estimation of divergence-based sensitivity indices with Gaussian process surrogates
• Fourier-domain transfer entropy spectrum
• Model-Driven Data Collection for Biological Systems
• Identification of Latent Variables From Graphical Model Residuals
• Estimating Conditional Mutual Information for Discrete-Continuous Mixtures using Multi-Dimensional Adaptive Histograms
• Smart Build Targets Batching Service at Google
• Privacy-Cost Management in Smart Meters with Mutual Information-Based Reinforcement Learning
• Causal conditioning and instantaneous coupling in causality graphs