Evaluating representations by the complexity of learning low-loss predictors

Abstract

We consider the problem of evaluating representations of data for use in solving a downstream task. We propose to measure the quality of a representation by the complexity of learning a predictor on top of the representation that achieves low loss on a task of interest, and introduce two methods, surplus description length (SDL) and $\epsilon$ sample complexity ($\epsilon$SC). In contrast to prior methods, which measure the amount of information about the optimal predictor that is present in a specific amount of data, our methods measure the amount of information needed from the data to recover an approximation of the optimal predictor up to a specified tolerance. We present a framework to compare these methods based on plotting the validation loss versus evaluation dataset size (the "loss-data" curve). Existing measures, such as mutual information and minimum description length probes, correspond to slices and integrals along the data axis of the loss-data curve, while ours correspond to slices and integrals along the loss axis. We provide experiments on real data to compare the behavior of each of these methods over datasets of varying size along with a high performance open source library for representation evaluation at https://github.com/willwhitney/reprieve.

References in corpus (15)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• A Simple Framework for Contrastive Learning of Visual Representations
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• A tutorial introduction to the minimum description length principle
• Learning and Evaluating General Linguistic Intelligence
• Understanding intermediate layers using linear classifier probes
• The Description Length of Deep Learning Models
• A Theory of Usable Information Under Computational Constraints
• Learning Optimal Representations with the Decodable Information Bottleneck
• Probing the State of the Art: A Critical Look at Visual Representation Evaluation

Cited by in corpus (4)

• Pareto Probing: Trading Off Accuracy for Complexity
• When Do You Need Billions of Words of Pretraining Data?
• Rissanen Data Analysis: Examining Dataset Characteristics via Description Length
• DirectProbe: Studying Representations without Classifiers