Assessing Phrasal Representation and Composition in Transformers
arXiv:2010.03763
Abstract
Deep transformer models have pushed performance on NLP tasks to new limits, suggesting sophisticated treatment of complex linguistic inputs, such as phrases. However, we have limited understanding of how these models handle representation of phrases, and whether this reflects sophisticated composition of phrase meaning like that done by humans. In this paper, we present systematic analysis of phrasal representations in state-of-the-art pre-trained transformers. We use tests leveraging human judgments of phrase similarity and meaning shift, and compare results before and after control of word overlap, to tease apart lexical effects versus composition effects. We find that phrase representation in these models relies heavily on word content, with little evidence of nuanced composition. We also identify variations in phrase representation quality across models, layers, and representation types, and make corresponding recommendations for usage of representations from these models.
arXiv commentAccepted at EMNLP 2020
References in corpus (7)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- What you can cram into a single vector: Probing sentence embeddings for linguistic properties
- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
- Memorize or generalize? Searching for a compositional RNN in a haystack
- Learning compositionally through attentive guidance
- SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics
- Does BERT agree? Evaluating knowledge of structure dependence through agreement relations