Assessing BERT's Syntactic Abilities

Abstract

I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.

Cited by in corpus (102)

• Pre-trained Models for Natural Language Processing: A Survey
• On the Cross-lingual Transferability of Monolingual Representations
• A Survey on Data Augmentation for Text Classification
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)
• Language Models as Knowledge Bases?
• Do Attention Heads in BERT Track Syntactic Dependencies?
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Universal Text Representation from BERT: An Empirical Study
• Is Multilingual BERT Fluent in Language Generation?
• Linguistic Profiling of a Neural Language Model
• Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• When BERT Plays the Lottery, All Tickets Are Winning
• Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
• Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning
• Measuring and Improving Consistency in Pretrained Language Models
• What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
• A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages
• Measuring Systematic Generalization in Neural Proof Generation with Transformers
• A Systematic Assessment of Syntactic Generalization in Neural Language Models
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Visualizing and Understanding the Effectiveness of BERT
• 75 Languages, 1 Model: Parsing Universal Dependencies Universally
• Contextual Word Representations: A Contextual Introduction
• Probing Neural Language Models for Human Tacit Assumptions
• Does injecting linguistic structure into language models lead to better alignment with brain recordings?
• Transformers: "The End of History" for NLP?
• Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
• Open Sesame: Getting Inside BERT's Linguistic Knowledge
• Compositionality decomposed: how do neural networks generalise?
• Pre-Trained Models: Past, Present and Future
• On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification
• Distilling Relation Embeddings from Pre-trained Language Models
• Syntactic Data Augmentation Increases Robustness to Inference Heuristics
• Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
• Can RNNs learn Recursive Nested Subject-Verb Agreements?
• Does Data Augmentation Improve Generalization in NLP?
• Topic Modeling with Contextualized Word Representation Clusters
• The Life Cycle of Knowledge in Big Language Models: A Survey
• Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
• Transformers Generalize Linearly
• Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter
• On the proper role of linguistically-oriented deep net analysis in linguistic theorizing
• Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
• Cold-start Active Learning through Self-supervised Language Modeling
• Parsing as Pretraining
• Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
• Improving AMR Parsing with Sequence-to-Sequence Pre-training
• Which *BERT? A Survey Organizing Contextualized Encoders
• Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?
• A Framework for Evaluation of Machine Reading Comprehension Gold Standards
• Question Generation from Paragraphs: A Tale of Two Hierarchical Models
• Natural Language Inference in Context -- Investigating Contextual Reasoning over Long Texts
• Scalable Syntax-Aware Language Models Using Knowledge Distillation
• On the Robustness of Language Encoders against Grammatical Errors
• How Does Adversarial Fine-Tuning Benefit BERT?
• Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
• BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?
• Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads
• My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
• Data-driven models and computational tools for neurolinguistics: a language technology perspective
• Lessons Learned from Applying off-the-shelf BERT: There is no Silver Bullet
• Neural Machine Translation: A Review and Survey
• BERTnesia: Investigating the capture and forgetting of knowledge in BERT
• Improving BERT Pretraining with Syntactic Supervision
• Weakly-Supervised Neural Response Selection from an Ensemble of Task-Specialised Dialogue Agents
• Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing
• On the Copying Behaviors of Pre-Training for Neural Machine Translation
• Recent Advances in Automated Question Answering In Biomedical Domain
• Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
• Intrinsic Knowledge Evaluation on Chinese Language Models
• Multi-Stream Transformers
• On Commonsense Cues in BERT for Solving Commonsense Tasks
• Improving Similar Language Translation With Transfer Learning
• Frequency Effects on Syntactic Rule Learning in Transformers
• A Relation-Oriented Clustering Method for Open Relation Extraction
• Are Transformers a Modern Version of ELIZA? Observations on French Object Verb Agreement
• Analysing the Effect of Masking Length Distribution of MLM: An Evaluation Framework and Case Study on Chinese MRC Datasets
• Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans
• Specializing Word Embeddings (for Parsing) by Information Bottleneck
• Word Frequency Does Not Predict Grammatical Knowledge in Language Models
• Sensing Ambiguity in Henry James' "The Turn of the Screw"
• Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models
• Probing for Bridging Inference in Transformer Language Models
• A Generative Approach to Titling and Clustering Wikipedia Sections
• Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus
• Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction
• Exploring the Role of BERT Token Representations to Explain Sentence Probing Results
• Variation and generality in encoding of syntactic anomaly information in sentence embeddings
• Have Attention Heads in BERT Learned Constituency Grammar?
• Interpreting Language Models Through Knowledge Graph Extraction
• Influence Patterns for Explaining Information Flow in BERT
• Learning Syntactic Dense Embedding with Correlation Graph for Automatic Readability Assessment
• IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize?
• Does Chinese BERT Encode Word Structure?
• Structured Self-Attention Weights Encode Semantics in Sentiment Analysis