On the Interplay Between Fine-tuning and Composition in Transformers
arXiv:2105.14668
Abstract
Pre-trained transformer language models have shown remarkable performance on a variety of NLP tasks. However, recent research has suggested that phrase-level representations in these models reflect heavy influences of lexical content, but lack evidence of sophisticated, compositional phrase information. Here we investigate the impact of fine-tuning on the capacity of contextualized embeddings to capture phrase meaning information beyond lexical content. Specifically, we fine-tune models on an adversarial paraphrase classification task with high lexical overlap, and on a sentiment classification task. After fine-tuning, we analyze phrasal representations in controlled settings following prior work. We find that fine-tuning largely fails to benefit compositionality in these representations, though training on sentiment yields a small, localized benefit for certain models. In follow-up analyses, we identify confounding cues in the paraphrase dataset that may explain the lack of composition benefits from that task, and we discuss potential factors underlying the localized benefits from sentiment training.
arXiv commentTo appear in Findings of ACL 2021
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
- BERT-ATTACK: Adversarial Attack Against BERT Using BERT
- Memorize or generalize? Searching for a compositional RNN in a haystack
- Compositional Explanations of Neurons
- Linguistic Profiling of a Neural Language Model
- Probing Neural Network Comprehension of Natural Language Arguments
- Measuring Compositionality in Representation Learning
- Learning compositionally through attentive guidance
- Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
- Assessing Composition in Sentence Vector Representations
- Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining