Inducing Syntactic Trees from BERT Representations
arXiv:1906.11511
Abstract
We use the English model of BERT and explore how a deletion of one word in a sentence changes representations of other words. Our hypothesis is that removing a reducible word (e.g. an adjective) does not affect the representation of other words so much as removing e.g. the main verb, which makes the sentence ungrammatical and of "high surprise" for the language model. We estimate reducibilities of individual words and also of longer continuous phrases (word n-grams), study their syntax-related properties, and then also use them to induce full dependency trees.
Accepted abstract for the BlackboxNLP 2019
Cited by in corpus (7)
- Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
- Interpreting Deep Learning Models in Natural Language Processing: A Review
- Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
- Pre-Trained Models: Past, Present and Future
- Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
- Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
- Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models