Assessing BERT's Syntactic Abilities

Abstract

I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.

Cited by in corpus (5)

• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Open Sesame: Getting Inside BERT's Linguistic Knowledge
• Scalable Syntax-Aware Language Models Using Knowledge Distillation