Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
arXiv:1908.09892
Abstract
Learning representations that accurately model semantics is an important goal of natural language processing research. Many semantic phenomena depend on syntactic structure. Recent work examines the extent to which state-of-the-art models for pre-training representations, such as BERT, capture such structure-dependent phenomena, but is largely restricted to one phenomenon in English: number agreement between subjects and verbs. We evaluate BERT's sensitivity to four types of structure-dependent agreement relations in a new semi-automatically curated dataset across 26 languages. We show that both the single-language and multilingual BERT models capture syntax-sensitive agreement patterns well in general, but we also highlight the specific linguistic contexts in which their performance degrades.
arXiv comment6 pages
References in corpus (5)
- Skip-Thought Vectors
- Assessing BERT's Syntactic Abilities
- Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks
- DisSent: Sentence Representation Learning from Explicit Discourse Relations
- Open Sesame: Getting Inside BERT's Linguistic Knowledge
Cited by in corpus (6)
- Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
- Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
- Assessing Phrasal Representation and Composition in Transformers
- On the Interplay Between Fine-tuning and Composition in Transformers
- Evaluating German Transformer Language Models with Syntactic Agreement Tests
- Recurrent babbling: evaluating the acquisition of grammar from limited input data