The Description Length of Deep Learning Models

Abstract

Solomonoff's general theory of inference and the Minimum Description Length principle formalize Occam's razor, and hold that a good model of data is a model that is good at losslessly compressing the data, including the cost of describing the model itself. Deep neural networks might seem to go against this principle given the large number of parameters to be encoded. We demonstrate experimentally the ability of deep neural networks to compress the training data even when accounting for parameter encoding. The compression viewpoint originally motivated the use of variational methods in neural networks. Unexpectedly, we found that these variational methods provide surprisingly poor compression bounds, despite being explicitly built to minimize such bounds. This might explain the relatively poor practical performance of variational methods in deep learning. On the other hand, simple incremental encoding methods yield excellent compression values on deep networks, vindicating Solomonoff's approach.

arXiv commentNIPS 2018

Cited by in corpus (23)

• True Few-Shot Learning with Language Models
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Learning and Evaluating General Linguistic Intelligence
• Confidence Calibration for Convolutional Neural Networks Using Structured Dropout
• Pareto Probing: Trading Off Accuracy for Complexity
• What they do when in doubt: a study of inductive biases in seq2seq learners
• Sparsely Activated Networks
• Evaluating representations by the complexity of learning low-loss predictors
• Bayesian Inference with Certifiable Adversarial Robustness
• Improving Generalization by Controlling Label-Noise Information in Neural Network Weights
• What Is Considered Complete for Visual Recognition?
• Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks
• Rissanen Data Analysis: Examining Dataset Characteristics via Description Length
• Continual Learning from the Perspective of Compression
• Representation Edit Distance as a Measure of Novelty
• Revisiting minimum description length complexity in overparameterized models
• Neural Abstract Reasoner
• Real-Time Uncertainty Estimation in Computer Vision via Uncertainty-Aware Distribution Distillation
• OSOA: One-Shot Online Adaptation of Deep Generative Models for Lossless Compression
• Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation
• Prequential MDL for Causal Structure Learning with Neural Networks
• Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP
• Measuring Information Transfer in Neural Networks