Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
arXiv:2109.14017 · doi:10.18653/v1/2021.mrl-1.17
Abstract
Recent research has adopted a new experimental field centered around the concept of text perturbations which has revealed that shuffled word order has little to no impact on the downstream performance of Transformer-based language models across many NLP tasks. These findings contradict the common understanding of how the models encode hierarchical and structural information and even question if the word order is modeled with position embeddings. To this end, this paper proposes nine probing datasets organized by the type of \emph{controllable} text perturbation for three Indo-European languages with a varying degree of word order flexibility: English, Swedish and Russian. Based on the probing analysis of the M-BERT and M-BART models, we report that the syntactic sensitivity depends on the language and model pre-training objectives. We also find that the sensitivity grows across layers together with the increase of the perturbation granularity. Last but not least, we show that the models barely use the positional information to induce syntactic trees from their intermediate self-attention and contextualized representations.
accepted to MRL @ EMNLP 2021
References in corpus (12)
- Multilingual Denoising Pre-training for Neural Machine Translation
- Assessing BERT's Syntactic Abilities
- StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
- Do Attention Heads in BERT Track Syntactic Dependencies?
- Linguistic Profiling of a Neural Language Model
- What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
- On the Importance of Word Order Information in Cross-lingual Sequence Labeling
- Inducing Syntactic Trees from BERT Representations
- DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing
- Parsing as Pretraining
- What Context Features Can Transformer Language Models Use?
- Data Augmentation via Dependency Tree Morphing for Low-Resource Languages