Rissanen Data Analysis: Examining Dataset Characteristics via Description Length
arXiv:2103.03872
Abstract
We introduce a method to determine if a certain capability helps to achieve an accurate model of given data. We view labels as being generated from the inputs by a program composed of subroutines with different capabilities, and we posit that a subroutine is useful if and only if the minimal program that invokes it is shorter than the one that does not. Since minimum program length is uncomputable, we instead estimate the labels' minimum description length (MDL) as a proxy, giving us a theoretically-grounded method for analyzing dataset characteristics. We call the method Rissanen Data Analysis (RDA) after the father of MDL, and we showcase its applicability on a wide variety of settings in NLP, ranging from evaluating the utility of generating subquestions before answering a question, to analyzing the value of rationales and explanations, to investigating the importance of different parts of speech, and uncovering dataset gender bias.
arXiv commentCode at https://github.com/ethanjperez/rda along with a script to run RDA on your own dataset
References in corpus (12)
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Semantics derived automatically from language corpora contain human-like biases
- Longformer: The Long-Document Transformer
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
- Understanding Black-box Predictions via Influence Functions
- e-SNLI: Natural Language Inference with Natural Language Explanations
- A tutorial introduction to the minimum description length principle
- Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment
- Understanding intermediate layers using linear classifier probes
- Understanding the Origins of Bias in Word Embeddings
- The Description Length of Deep Learning Models
- Evaluating representations by the complexity of learning low-loss predictors