Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
arXiv:2002.00737
Abstract
With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.
arXiv commentICLR 2020
References in corpus (7)
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Assessing BERT's Syntactic Abilities
- Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
- Visually Grounded Neural Syntax Acquisition
- Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders
- A Critical Analysis of Biased Parsers in Unsupervised Parsing
- Inducing Syntactic Trees from BERT Representations
Cited by in corpus (13)
- Pre-trained Models for Natural Language Processing: A Survey
- Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
- A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages
- Probing BERT in Hyperbolic Spaces
- Linguistic Dependencies and Statistical Dependence
- Neural Bi-Lexicalized PCFG Induction
- Improving BERT Pretraining with Syntactic Supervision
- Syntax Representation in Word Embeddings and Neural Networks -- A Survey
- On the Branching Bias of Syntax Extracted from Pre-trained Language Models
- Unsupervised Parsing via Constituency Tests
- IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize?
- Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models
- Unsupervised Full Constituency Parsing with Neighboring Distribution Divergence