Linguistic Dependencies and Statistical Dependence

Abstract

Are pairs of words that tend to occur together also likely to stand in a linguistic dependency? This empirical question is motivated by a long history of literature in cognitive science, psycholinguistics, and NLP. In this work we contribute an extensive analysis of the relationship between linguistic dependencies and statistical dependence between words. Improving on previous work, we introduce the use of large pretrained language models to compute contextualized estimates of the pointwise mutual information between words (CPMI). For multiple models and languages, we extract dependency trees which maximize CPMI, and compare to gold standard linguistic dependencies. Overall, we find that CPMI dependencies achieve an unlabelled undirected attachment score of at most $\approx 0.5$. While far above chance, and consistently above a non-contextualized PMI baseline, this score is generally comparable to a simple baseline formed by connecting adjacent words. We analyze which kinds of linguistic dependencies are best captured in CPMI dependencies, and also find marked differences between the estimates of the large pretrained language models, illustrating how their different training schemes affect the type of dependencies they capture.

arXiv commentEMNLP2021 camera-ready version. 9 pages, plus references and appendices

References in corpus (10)

• Distributed Representations of Words and Phrases and their Compositionality
• The information bottleneck method
• Cross-lingual Language Model Pretraining
• Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection
• Discovery of Linguistic Relations Using Lexical Attraction
• Do Attention Heads in BERT Track Syntactic Dependencies?
• Analogies Explained: Towards Understanding Word Embeddings
• An Empirical Comparison of Probability Models for Dependency Grammar
• Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
• Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics