Pareto Probing: Trading Off Accuracy for Complexity
arXiv:2010.02180 · doi:10.18653/v1/2020.emnlp-main.254
Abstract
The question of how to probe contextual word representations for linguistic structure in a way that is both principled and useful has seen significant attention recently in the NLP literature. In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume. To measure complexity, we present a number of parametric and non-parametric metrics. Our experiments using Pareto hypervolume as an evaluation metric show that probes often do not conform to our expectations -- e.g., why should the non-contextual fastText representations encode more morpho-syntactic information than the contextual BERT representations? These results suggest that common, simplistic probing tasks, such as part-of-speech labeling and dependency arc labeling, are inadequate to evaluate the linguistic structure encoded in contextual word representations. This leads us to propose full dependency parsing as a probing task. In support of our suggestion that harder probing tasks are necessary, our experiments with dependency parsing reveal a wide gap in syntactic knowledge between contextual and non-contextual representations.
arXiv commentTiago Pimentel and Naomi Saphra contributed equally to this work. Camera ready version of EMNLP 2020 publication. In this new version, we fixed some notation issues in the appendix, and added a new appendix section describing our MLP. Code available in https://github.com/rycolab/pareto-probing
References in corpus (5)
- Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
- Understanding intermediate layers using linear classifier probes
- The Description Length of Deep Learning Models
- Evaluating representations by the complexity of learning low-loss predictors
- On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networks