The information bottleneck method

Abstract

We define the relevant information in a signal $x\in X$ as being the information that this signal provides about another signal $y\in \Y$. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal $x$ requires more than just predicting $y$, it also requires specifying which features of $\X$ play a role in the prediction. We formalize this problem as that of finding a short code for $\X$ that preserves the maximum information about $\Y$. That is, we squeeze the information that $\X$ provides about $\Y$ through a `bottleneck' formed by a limited set of codewords $\tX$. This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure $d(x,\x)$ emerges from the joint statistics of $\X$ and $\Y$. This approach yields an exact set of self consistent equations for the coding rules $X \to \tX$ and $\tX \to \Y$. Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.

Cited by in corpus (449)

• Community detection in graphs
• A Review of Feature Selection Methods Based on Mutual Information
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Opening the Black Box of Deep Neural Networks via Information
• What Makes for Good Views for Contrastive Learning?
• Discovering physical concepts with neural networks
• Data Mining and Machine Learning in Astronomy
• Deep Variational Information Bottleneck
• A Survey on Digital Twins: Architecture, Enabling Technologies, Security and Privacy, and Future Prospects
• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
• Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
• Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with NBEATSx
• Predictability and hierarchy in Drosophila behavior
• A Bivariate Measure of Redundant Information
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Mutual Information, Neural Networks and the Renormalization Group
• Semantic information, autonomous agency, and nonequilibrium statistical physics
• The Revisiting Problem in Simultaneous Localization and Mapping: A Survey on Visual Loop Closure Detection
• Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations
• The free energy principle made simpler but not too simple
• Generalization in Quantum Machine Learning: a Quantum Information Perspective
• Nonlinear Information Bottleneck
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Entropy and mutual information in models of deep neural networks
• Evolution of bow-tie architectures in biology
• Dream to Control: Learning Behaviors by Latent Imagination
• Compressing Neural Networks using the Variational Information Bottleneck
• Graph Barlow Twins: A self-supervised representation learning framework for graphs
• Self-Supervised Information Bottleneck for Deep Multi-View Subspace Clustering
• Invariant Representations without Adversarial Training
• Learning Disentangled Representations in the Imaging Domain
• The Conditional Entropy Bottleneck
• Machine Unlearning: Solutions and Challenges
• Graph Information Bottleneck
• A survey on intrinsic motivation in reinforcement learning
• Contrastive Collaborative Filtering for Cold-Start Item Recommendation
• Learning Robust Representations via Multi-View Information Bottleneck
• Empirical Bayes Transductive Meta-Learning with Synthetic Gradients
• Mathematics of Deep Learning
• Decoding Photons: Physics in the Latent Space of a BIB-AE Generative Network
• Information theoretic approach to interactive learning
• Meta-Learning without Memorization
• GATE: Graph CCA for Temporal SElf-supervised Learning for Label-efficient fMRI Analysis
• Discovering Structure in High-Dimensional Data Through Correlation Explanation
• Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness
• MSTREAM: Fast Anomaly Detection in Multi-Aspect Streams
• Semantic Invariant Multi-view Clustering with Fully Incomplete Information
• Formal Limitations on the Measurement of Mutual Information
• Relevant sparse codes with variational information bottleneck
• An evaluation of intrusive instrumental intelligibility metrics
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Modeling Uncertainty with Hedged Instance Embedding
• Restricting the Flow: Information Bottlenecks for Attribution
• Semantic categories of artifacts and animals reflect efficient coding
• Statistical Criticality arises in Most Informative Representations
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• Combining pretrained CNN feature extractors to enhance clustering of complex natural images
• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• Relative Information Loss in the PCA
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• Efficient Saliency Maps for Explainable AI
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• An Unsupervised Autoregressive Model for Speech Representation Learning
• Where is the Information in a Deep Neural Network?
• Information-Theoretic Caching: Sequential Coding for Computing
• Predicting the Popularity of Micro-videos with Multimodal Variational Encoder-Decoder Framework
• Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization
• InfoBot: Transfer and Exploration via the Information Bottleneck
• Prioritized Information Bottleneck Theoretic Framework with Distributed Online Learning for Edge Video Analytics
• Towards Understanding Sparse Filtering: A Theoretical Perspective
• Robust Training of Vector Quantized Bottleneck Models
• Thermodynamic cost and benefit of memory
• A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime
• Fixing a Broken ELBO
• Multivariate Approaches to Classification in Extragalactic Astronomy
• Learning Efficient Multi-agent Communication: An Information Bottleneck Approach
• The evolution of lossy compression
• Steps Towards a Theory of Visual Information: Active Perception, Signal-to-Symbol Conversion and the Interplay Between Sensing and Control
• Distributed Information-Theoretic Clustering
• Information Bottlenecks, Causal States, and Statistical Relevance Bases: How to Represent Relevant Information in Memoryless Transduction
• Radioactive data: tracing through training
• Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs
• Trading bits in the readout from a genetic network
• Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction
• InfoGCL: Information-Aware Graph Contrastive Learning
• Informationally restricted quantum correlations
• Systematic identification of statistically significant network measures
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders
• Bayesian model and dimension reduction for uncertainty propagation: applications in random media
• Dual Semantic Fusion Network for Video Object Detection
• Deep Semi-Supervised Anomaly Detection
• What-is and How-to for Fairness in Machine Learning: A Survey, Reflection, and Perspective
• Information Theoretic Counterfactual Learning from Missing-Not-At-Random Feedback
• Convexity and Operational Interpretation of the Quantum Information Bottleneck Function
• Visual Representations: Defining Properties and Deep Approximations
• A probabilistic generative model for semi-supervised training of coarse-grained surrogates and enforcing physical constraints through virtual observables
• Architectural Vision for Quantum Computing in the Edge-Cloud Continuum
• Deep radiomic signature with immune cell markers predicts the survival of glioma patients
• Maximally Informative Hierarchical Representations of High-Dimensional Data
• 3DMolNet: A Generative Network for Molecular Structures
• Advances in Variational Inference
• The Minimum Description Length Principle for Pattern Mining: A Survey
• Shannon Meets Carnot: Generalized Second Thermodynamic Law
• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
• L2/L2-foreach sparse recovery with low risk
• Significance-aware Information Bottleneck for Domain Adaptive Semantic Segmentation
• Quantum Rate-Distortion Coding of Relevant Information
• Quantifying Relevance in Learning and Inference
• Information bottleneck through variational glasses
• An information theoretic approach to the functional classification of neurons
• Anti-Spoofing Using Transfer Learning with Variational Information Bottleneck
• Distributed Compression of Correlated Classical-Quantum Sources or: The Price of Ignorance
• Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation
• Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization
• Improving the Adversarial Robustness of NLP Models by Information Bottleneck
• Information-theoretic analysis of the directional influence between cellular processes
• Semi-supervised cross-entropy clustering with information bottleneck constraint
• Relevance in the Renormalization Group and in Information Theory
• Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples
• Adversarial Neural Pruning with Latent Vulnerability Suppression
• TherML: Thermodynamics of Machine Learning
• Bayesian Renormalization
• Learning Optimal Representations with the Decodable Information Bottleneck
• Bi-fidelity Variational Auto-encoder for Uncertainty Quantification
• Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients
• Limits on Inferring T-cell Specificity from Partial Information
• High-Fidelity GAN Inversion for Image Attribute Editing
• Caveats for information bottleneck in deterministic scenarios
• Optimal high-level descriptions of dynamical systems
• D-Lite: Navigation-Oriented Compression of 3D Scene Graphs for Multi-Robot Collaboration
• Conditional Mutual Information Neural Estimator
• Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks
• Machines Learn to Infer Stellar Parameters Just by Looking at a Large Number of Spectra
• InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle
• Graph-based Predictable Feature Analysis
• FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing
• Contrastive learning, multi-view redundancy, and linear models
• Phase Transitions for the Information Bottleneck in Representation Learning
• Reasoning About Generalization via Conditional Mutual Information
• A framework for redescription set construction
• Efficient algorithms for quantum information bottleneck
• Dynamics-aware Embeddings
• An Information-Theoretic Optimality Principle for Deep Reinforcement Learning
• Implementing quantum dimensionality reduction for non-Markovian stochastic simulation
• Information Dropout: Learning Optimal Representations Through Noisy Computation
• Large Language Models at Work in China's Labor Market
• A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning
• Information theory and learning: a physical approach
• Information flows of diverse autoencoders
• A Generalized Framework for Kullback-Leibler Markov Aggregation
• Dynamic Bottleneck for Robust Self-Supervised Exploration
• Unified Adversarial Invariance
• Hierarchical State Abstractions for Decision-Making Problems with Computational Constraints
• Signal Enhancement as Minimization of Relevant Information Loss
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
• An Identifiable Double VAE For Disentangled Representations
• Layer-wise Characterization of Latent Information Leakage in Federated Learning
• Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration
• Towards More Human-like AI Communication: A Review of Emergent Communication Research
• Learning to Learn with Variational Information Bottleneck for Domain Generalization
• PAC-Bayes Information Bottleneck
• Information Loss in Static Nonlinearities
• Inflation as an Information Bottleneck - A strategy for identifying universality classes and making robust predictions
• Enforcing Encoder-Decoder Modularity in Sequence-to-Sequence Models
• Collaborative Information Bottleneck
• Learning in embodied action-perception loops through exploration
• Self-supervised Learning from a Multi-view Perspective
• Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification
• Diversity inducing Information Bottleneck in Model Ensembles
• Adversarially Regularized Graph Attention Networks for Inductive Learning on Partially Labeled Graphs
• Predictive Uncertainty Quantification with Compound Density Networks
• An Information Theory-inspired Strategy for Automatic Network Pruning
• Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
• An Information-Theoretic Formalism for Multiscale Structure in Complex Systems
• Variational Predictive Information Bottleneck
• MemDA: Forecasting Urban Time Series with Memory-based Drift Adaptation
• Improving Unsupervised Domain Adaptation with Variational Information Bottleneck
• Structure or Noise?
• Divergence and Sufficiency for Convex Optimization
• Maximum Entropy and Sufficiency
• Fundamental Limits and Tradeoffs in Invariant Representation Learning
• A Survey on Deep-Learning based Techniques for Modeling and Estimation of MassiveMIMO Channels
• Information Laundering for Model Privacy
• Compression-Based Regularization with an Application to Multi-Task Learning
• Efficient human-like semantic representations via the Information Bottleneck principle
• Linguistic Dependencies and Statistical Dependence
• Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
• Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysis
• Rationalizing Predictions by Adversarial Information Calibration
• Learning Robust Representation through Graph Adversarial Contrastive Learning
• Neural Distributed Compressor Discovers Binning
• Flow of Information in Feed-Forward Deep Neural Networks
• Flow Matching for Optimal Reaction Coordinates of Biomolecular System
• Minimizing Impurity Partition Under Constraints
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• Detecting Beneficial Feature Interactions for Recommender Systems
• Contrastive variational information bottleneck for aspect-based sentiment analysis
• On the Maximum Mutual Information Capacity of Neural Architectures
• Information Obfuscation of Graph Neural Networks
• Quantum entropy couples matter with geometry
• Information Theoretic Co-Training
• Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness
• DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning
• Information Losses in Neural Classifiers from Sampling
• Set-based complexity and biological information
• Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration
• Learning Discrete Structured Representations by Adversarially Maximizing Mutual Information
• Adversarial Learning of Privacy-Preserving and Task-Oriented Representations
• Unsupervised Hashing with Contrastive Information Bottleneck
• Renormalized Mutual Information for Artificial Scientific Discovery
• Synergy, Redundancy and Common Information
• To Split or Not to Split: The Impact of Disparate Treatment in Classification
• A Submodularity-based Agglomerative Clustering Algorithm for the Privacy Funnel
• A new keyphrases extraction method based on suffix tree data structure for arabic documents clustering
• Mutual Information Gradient Estimation for Representation Learning
• Misanthropic Entropy and Renormalization as a Communication Channel
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model
• Novelty Search in Representational Space for Sample Efficient Exploration
• Information Potential Auto-Encoders
• Recognizing Predictive Substructures with Subgraph Information Bottleneck
• Privileged Information Dropout in Reinforcement Learning
• Fate of Duplicated Neural Structures
• Variational Leakage: The Role of Information Complexity in Privacy Leakage
• Information Bottleneck Theory on Convolutional Neural Networks
• Theoretical Analysis of Deep Neural Networks in Physical Layer Communication
• The Role of Information Complexity and Randomization in Representation Learning
• Dynamic learning rate using Mutual Information
• Learning Discrete State Abstractions With Deep Variational Inference
• Towards Empirical Sandwich Bounds on the Rate-Distortion Function
• Partially Observable Szilard Engines
• Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning
• InfoShape: Task-Based Neural Data Shaping via Mutual Information
• Information-Theoretic Abstractions for Resource-Constrained Agents via Mixed-Integer Linear Programming
• Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification
• Embarrassingly Simple Binary Representation Learning
• Iterative minimization algorithm on a mixture family
• Generalized quantum Arimoto-Blahut algorithm and its application to quantum information bottleneck
• Which Boolean Functions are Most Informative?
• Pareto-optimal clustering with the primal deterministic information bottleneck
• The Minimum Information Principle for Discriminative Learning
• Gaussian Lower Bound for the Information Bottleneck Limit
• Action-Sufficient State Representation Learning for Control with Structural Constraints
• Probabilistic Reduced-Order Modeling for Stochastic Partial Differential Equations
• A Rate-Distortion Framework for Characterizing Semantic Information
• A Survey of Exploration Methods in Reinforcement Learning
• Shredder: Learning Noise Distributions to Protect Inference Privacy
• Observing Schrödinger's Cat with Artificial Intelligence: Emergent Classicality from Information Bottleneck
• Multiscale Principle of Relevant Information for Hyperspectral Image Classification
• Robust Privatization with Multiple Tasks and the Optimal Privacy-Utility Tradeoff
• Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers
• Data Sanitisation Protocols for the Privacy Funnel with Differential Privacy Guarantees
• Information Theoretic Meta Learning with Gaussian Processes
• Information Bottleneck Constrained Latent Bidirectional Embedding for Zero-Shot Learning
• From Quantum Source Compression to Quantum Thermodynamics
• Differentiable Joint Pruning and Quantization for Hardware Efficiency
• Multi-Task Variational Information Bottleneck
• On Predictive Information in RNNs
• Unsupervised model-free representation learning
• Critical Slowing Down Near Topological Transitions in Rate-Distortion Problems
• Cultural evolution via iterated learning and communication explains efficient color naming systems
• Fractionally Predictive Spiking Neurons
• Variational Information Bottleneck on Vector Quantized Autoencoders
• InfoFair: Information-Theoretic Intersectional Fairness
• General Information Bottleneck Objectives and their Applications to Machine Learning
• EASE: Extractive-Abstractive Summarization with Explanations
• Ambitions for theory in the physics of life
• Subjective Reality and Strong Artificial Intelligence
• What Information Does a ResNet Compress?
• On the Estimation of Information Measures of Continuous Distributions
• Meta Dropout: Learning to Perturb Features for Generalization
• Abstraction in decision-makers with limited information processing capabilities
• Predictive information in a sensory population
• Entropy Minimization In Emergent Languages
• Improving Disentangled Text Representation Learning with Information-Theoretic Guidance
• MetaInfoNet: Learning Task-Guided Information for Sample Reweighting
• Learning from the Best: Rationalizing Prediction by Adversarial Information Calibration
• Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training
• Mutual information estimation for graph convolutional neural networks
• Robust and Explainable Autoencoders for Unsupervised Time Series Outlier Detection---Extended Version
• Three-body renormalization group limit cycles based on unsupervised feature learning
• Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• An Empirical Study of Graph Contrastive Learning
• Graph Information Bottleneck for Subgraph Recognition
• A Bayesian Evaluation Framework for Subjectively Annotated Visual Recognition Tasks
• Variational Representations and Neural Network Estimation of Rényi Divergences
• Large scale evaluation of importance maps in automatic speech recognition
• A synapse-centric account of the free energy principle
• Batch norm with entropic regularization turns deterministic autoencoders into generative models
• A Modular Deep Learning Pipeline for Galaxy-Scale Strong Gravitational Lens Detection and Modeling
• Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning
• Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck
• IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL
• Generative Adversarial Networks are Special Cases of Artificial Curiosity (1990) and also Closely Related to Predictability Minimization (1991)
• Learning Representations in Reinforcement Learning:An Information Bottleneck Approach
• Bregman Divergence Bounds and Universality Properties of the Logarithmic Loss
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• Multi-view Information-theoretic Co-clustering for Co-occurrence Data
• M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention
• Information Bottleneck in Control Tasks with Recurrent Spiking Neural Networks
• Hard Clusters Maximize Mutual Information
• Malleable Coding with Fixed Reuse
• Learning Curves for Mutual Information Maximization
• Learning to Find Correlated Features by Maximizing Information Flow in Convolutional Neural Networks
• Cooperative Training and Latent Space Data Augmentation for Robust Medical Image Segmentation
• Balancing New Against Old Information: The Role of Surprise in Learning
• Information-Theoretic Odometry Learning
• Robust Representation Learning via Perceptual Similarity Metrics
• ST-MAML: A Stochastic-Task based Method for Task-Heterogeneous Meta-Learning
• Unsupervised Explanation Generation for Machine Reading Comprehension
• MIC: Mutual Information based hierarchical Clustering
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• Do Compressed Representations Generalize Better?
• To Beta or Not To Beta: Information Bottleneck for DigitaL Image Forensics
• Principal Inertia Components and Applications
• Causal learning with sufficient statistics: an information bottleneck approach
• Drill the Cork of Information Bottleneck by Inputting the Most Important Data
• Joint Quantizer Optimization based on Neural Quantizer for Sum-Product Decoder
• A Theory of Feature Learning
• Deep Information Networks
• A Simple Hierarchical Pooling Data Structure for Loop Closure
• Variational Encoders and Autoencoders : Information-theoretic Inference and Closed-form Solutions
• What makes it possible to learn probability distributions in the natural world?
• Variational Information Bottleneck for Effective Low-resource Audio Classification
• Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
• A Free-Energy Principle for Representation Learning
• MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architectures
• Quantization of Binary-Input Discrete Memoryless Channels
• Evaluation of Dataflow through layers of Deep Neural Networks in Classification and Regression Problems
• Bounds on mutual information of mixture data for classification tasks
• Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environment
• Blockchain-Enabled Variational Information Bottleneck for IoT Networks
• Predictability, complexity and learning
• Information Plane Analysis Visualization in Deep Learning via Transfer Entropy
• A Provably Convergent Information Bottleneck Solution via ADMM
• Impact of Bit Allocation Strategies on Machine Learning Performance in Rate Limited Systems
• Compressive Visual Representations
• Rate-Regularization and Generalization in VAEs
• Quantifying multivariate redundancy with maximum entropy decompositions of mutual information
• Unrolled and Pipelined Decoders based on Look-Up Tables for Polar Codes
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• Higher-Order Kullback-Leibler Aggregation of Markov Chains
• Information Bottleneck: Exact Analysis of (Quantized) Neural Networks
• An Information-Theoretic Approach for Automatically Determining the Number of States when Aggregating Markov Chains
• Predictive Uncertainty through Quantization
• A Segmentation-Oriented Inter-Class Transfer Method: Application to Retinal Vessel Segmentation
• Separation of time scales and direct computation of weights in deep neural networks
• Scientists in silico?
• Minimum Rates of Approximate Sufficient Statistics
• Generalized Statistics Framework for Rate Distortion Theory with Bregman Divergences
• The Multi-layer Information Bottleneck Problem
• Learning Robust Variational Information Bottleneck with Reference
• Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy
• Optimal prediction and natural scene statistics in the retina
• Some new insights into information decomposition in complex systems based on common information
• The color of smiling: computational synaesthesia of facial expressions
• Developing Non-Stochastic Privacy-Preserving Policies Using Agglomerative Clustering
• Learning Conditional Invariance through Cycle Consistency
• Decomposing Representations for Deterministic Uncertainty Estimation
• Information Measures for Deterministic Input-Output Systems
• Understanding Neural Networks with Logarithm Determinant Entropy Estimator
• The physical language of molecular codes: A rate-distortion approach to the evolution and emergence of biological codes
• Predictive Overlapping Co-Clustering
• Test Sample Accuracy Scales with Training Sample Density in Neural Networks
• On the Design and Analysis of Multiple View Descriptors
• Compressed Hierarchical Representations for Multi-Task Learning and Task Clustering
• An Evidential Interpretation of the 1st and 2nd Laws of Thermodynamics
• Information based clustering: Supplementary material
• Information Bottleneck for an Oblivious Relay with Channel State Information: the Vector Case
• Margin Maximization as Lossless Maximal Compression
• BiDet: An Efficient Binarized Object Detector
• The distance between the weights of the neural network is meaningful
• Policy Gradients Incorporating the Future
• Inverse Learning of Symmetries
• QuASE: Question-Answer Driven Sentence Encoding
• Quantifying many-body learning far from equilibrium with representation learning
• The Privacy-Utility Tradeoff of Robust Local Differential Privacy
• A Deep, Information-theoretic Framework for Robust Biometric Recognition
• Information Theory-Guided Heuristic Progressive Multi-View Coding
• Verifiability and Predictability: Interpreting Utilities of Network Architectures for Point Cloud Processing
• A Non-equilibrium Thermodynamic Framework of Consciousness
• Variational Reward Estimator Bottleneck: Learning Robust Reward Estimator for Multi-Domain Task-Oriented Dialog
• On Controlled DeEntanglement for Natural Language Processing
• Analysis of Information Flow Through U-Nets
• What are Neural Networks made of?
• Nested Learning For Multi-Granular Tasks
• Filtering Statistics on Networks
• Deep Unsupervised Image Anomaly Detection: An Information Theoretic Framework
• From Artificial Intelligence to Brain Intelligence: The basis learning and memory algorithm for brain-like intelligence
• Specializing Word Embeddings (for Parsing) by Information Bottleneck
• Partial Information Decomposition of Boolean Functions: a Fourier Analysis perspective
• On the Relevance-Complexity Region of Scalable Information Bottleneck
• Inserting Information Bottlenecks for Attribution in Transformers
• Examining the causal structures of deep neural networks using information theory
• Neurons Activation Visualization and Information Theoretic Analysis
• On Learning Prediction-Focused Mixtures
• REVE: Regularizing Deep Learning with Variational Entropy Bound
• Towards Further Understanding of Sparse Filtering via Information Bottleneck
• Adversarial Examples as an Input-Fault Tolerance Problem
• Improving Supervised Phase Identification Through the Theory of Information Losses
• Probabilistic Clustering Using Maximal Matrix Norm Couplings
• A mathematical theory of imperfect communication: Energy efficiency considerations in multi-level coding
• Over-the-Air Multi-Sensor Inference with Neural Networks Using Memristor-Based Analog Computing
• Problems with information theoretic approaches to causal learning
• Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics
• Rediscovering Deep Neural Networks Through Finite-State Distributions
• Information-Bottleneck-Based Behavior Representation Learning for Multi-agent Reinforcement learning
• Quantifying the effect of representations on task complexity
• On the Information Theoretic Distance Measures and Bidirectional Helmholtz Machines
• Cause-Effect Deep Information Bottleneck For Systematically Missing Covariates
• Learning to Collaborate for User-Controlled Privacy
• From abstract items to latent spaces to observed data and back: Compositional Variational Auto-Encoder
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories
• Information Scaling Law of Deep Neural Networks
• Bounded rationality for relaxing best response and mutual consistency: The Quantal Hierarchy model of decision-making
• Applying the Information Bottleneck Principle to Prosodic Representation Learning
• Information Bottleneck Approach to Spatial Attention Learning
• Improving Generalization of Deep Networks for Inverse Reconstruction of Image Sequences
• WiSE-ALE: Wide Sample Estimator for Approximate Latent Embedding
• Privacy with Estimation Guarantees
• Analyzing neural responses to natural signals: Maximally informative dimensions
• When Stability meets Sufficiency: Informative Explanations that do not Overwhelm
• Resampling methods for document clustering
• Universality of Logarithmic Loss in Lossy Compression
• Ultimate Intelligence Part III: Measures of Intelligence, Perception and Intelligent Agents
• Learning disentangled representation for classical models
• Computation in a single neuron: Hodgkin and Huxley revisited
• Barlow Graph Auto-Encoder for Unsupervised Network Embedding
• Towards an Understanding of Default Policies in Multitask Policy Optimization
• On the Theory and Algorithm for rigorous discretization in applications of Information Theory
• Consensus Synergizes with Memory: A Simple Approach for Anomaly Segmentation in Urban Scenes
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Encoding Causal Macrovariables
• Ultimate Intelligence Part II: Physical Measure and Complexity of Intelligence
• Le Cam meets LeCun: Deficiency and Generic Feature Learning
• Nonasymptotic noisy lossy source coding
• Optimally fuzzy temporal memory
• A note on the unbiased estimation of mutual information
• Disentangled Variational Information Bottleneck for Multiview Representation Learning
• The Variational InfoMax Learning Objective
• Learning Transferable Concepts in Deep Reinforcement Learning
• Generalized Statistics Framework for Rate Distortion Theory
• An Information Bottleneck Problem with Rényi's Entropy
• Variational Information Bottleneck Model for Accurate Indoor Position Recognition
• On Equivariant and Invariant Learning of Object Landmark Representations
• Subgoal Planning Algorithm for Autonomous Vehicle Guidance