Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction

Abstract

With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.

ICLR 2020

References in corpus (3)

• Assessing BERT's Syntactic Abilities
• Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoders
• Inducing Syntactic Trees from BERT Representations

Cited by in corpus (5)

• A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages
• Probing BERT in Hyperbolic Spaces
• Syntax Representation in Word Embeddings and Neural Networks -- A Survey
• Improving BERT Pretraining with Syntactic Supervision
• IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize?