Black-box language model explanation by context length probing

Abstract

The increasingly widespread adoption of large language models has highlighted the need for improving their explainability. We present context length probing, a novel explanation technique for causal language models, based on tracking the predictions of a model as a function of the length of available context, and allowing to assign differential importance scores to different contexts. The technique is model-agnostic and does not rely on access to model internals beyond computing token-level probabilities. We apply context length probing to large pre-trained language models and offer some initial analyses and insights, including the potential for studying long-range dependencies. The source code and an interactive demo of the method are available.

arXiv comment11 pages, 9 figures. ACL 2023 short paper camera-ready. Demos at https://cifkao.github.io/context-probing/ and https://huggingface.co/spaces/cifkao/context-probing ; code at https://github.com/cifkao/context-probing/