TURBO: The Swiss Knife of Auto-Encoders
arXiv:2311.06527 · doi:10.3390/e25101471
Abstract
We present a novel information-theoretic framework, termed as TURBO, designed to systematically analyse and generalise auto-encoding methods. We start by examining the principles of information bottleneck and bottleneck-based networks in the auto-encoding setting and identifying their inherent limitations, which become more prominent for data with multiple relevant, physics-related representations. The TURBO framework is then introduced, providing a comprehensive derivation of its core concept consisting of the maximisation of mutual information between various data representations expressed in two directions reflecting the information flows. We illustrate that numerous prevalent neural network models are encompassed within this framework. The paper underscores the insufficiency of the information bottleneck concept in elucidating all such models, thereby establishing TURBO as a preferable theoretical reference. The introduction of TURBO contributes to a richer understanding of data representation and the structure of neural network models, enabling more efficient and versatile applications.
References in corpus (10)
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- Autoencoding beyond pixels using a learned similarity metric
- Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
- Getting High: High Fidelity Simulation of High Granularity Calorimeters with High Speed
- Invertible Networks or Partons to Detector and Back Again
- Learning to Simulate High Energy Particle Collisions from Unlabeled Data
- Information bottleneck through variational glasses
- Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding
- Turbo-Sim: a generalised generative model with a physical latent space