What Context Features Can Transformer Language Models Use?
arXiv:2106.08367
Abstract
Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model prediction? We describe a series of experiments that measure usable information by selectively ablating lexical and structural information in transformer language models trained on English Wikipedia. In both mid- and long-range contexts, we find that several extremely destructive context manipulations -- including shuffling word order within sentences and deleting all words other than nouns -- remove less than 15% of the usable information. Our results suggest that long contexts, but not their detailed syntactic and propositional content, are important for the low perplexity of current transformer language models.
14 pages, 7 figures, to be published at ACL 2021
References in corpus (8)
- Understanding deep learning requires rethinking generalization
- Linformer: Self-Attention with Linear Complexity
- Generating Long Sequences with Sparse Transformers
- Understanding Neural Networks through Representation Erasure
- Reformer: The Efficient Transformer
- Compressive Transformers for Long-Range Sequence Modelling
- A Theory of Usable Information Under Computational Constraints
- Rissanen Data Analysis: Examining Dataset Characteristics via Description Length