A tutorial introduction to the minimum description length principle

Abstract

This tutorial provides an overview of and introduction to Rissanen's Minimum Description Length (MDL) Principle. The first chapter provides a conceptual, entirely non-technical introduction to the subject. It serves as a basis for the technical introduction given in the second chapter, in which all the ideas of the first chapter are made mathematically precise. The main ideas are discussed in great conceptual and technical detail. This tutorial is an extended version of the first two chapters of the collection "Advances in Minimum Description Length: Theory and Application" (edited by P.Grunwald, I.J. Myung and M. Pitt, to be published by the MIT Press, Spring 2005).

80 pages 5 figures Report with 2 chapters

References in corpus (3)

• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• On Supervised Selection of Bayesian Networks
• Maximum Entropy and the Glasses You Are Looking Through

Cited by in corpus (31)

• True Few-Shot Learning with Language Models
• Finding large average submatrices in high dimensional data
• From task structures to world models: What do LLMs know?
• The statistical trade-off between word order and word structure - large-scale evidence for the principle of least effort
• MDL Denoising Revisited
• Minimum Description Length codes are critical
• What they do when in doubt: a study of inductive biases in seq2seq learners
• Data segmentation algorithms: Univariate mean change and beyond
• Evaluating representations by the complexity of learning low-loss predictors
• Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning
• Prediction with a Short Memory
• Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation
• Rissanen Data Analysis: Examining Dataset Characteristics via Description Length
• Minimum Description Length Principle for Maximum Entropy Model Selection
• Automatic Parameter Selection for Non-Redundant Clustering
• Edge based stochastic block model statistical inference
• EnergyNet: Energy-based Adaptive Structural Learning of Artificial Neural Network Architectures
• A Fast Synchronization Clustering Algorithm
• Rank Ordered Autoencoders
• Toward a Generalization Metric for Deep Generative Models
• Volumes of logistic regression models with applications to model selection
• Prequential MDL for Causal Structure Learning with Neural Networks
• GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL
• A Search for the Underlying Equation Governing Similar Systems
• Where Do Human Heuristics Come From?
• On the Estimation and Use of Statistical Modelling in Information Retrieval
• Inferring Latent dimension of Linear Dynamical System with Minimum Description Length
• The relativity of theory
• Higher-order asymptotics for the parametric complexity
• Synchronization Clustering based on a Linearized Version of Vicsek model
• Executable World Models for ARC-AGI-3 in the Era of Coding Agents