Emergent Properties of Finetuned Language Representation Models

Abstract

Large, self-supervised transformer-based language representation models have recently received significant amounts of attention, and have produced state-of-the-art results across a variety of tasks simply by scaling up pre-training on larger and larger corpora. Such models usually produce high dimensional vectors, on top of which additional task-specific layers and architectural modifications are added to adapt them to specific downstream tasks. Though there exists ample evidence that such models work well, we aim to understand what happens when they work well. We analyze the redundancy and location of information contained in output vectors for one such language representation model -- BERT. We show empirical evidence that the [CLS] embedding in BERT contains highly redundant information, and can be compressed with minimal loss of accuracy, especially for finetuned models, dovetailing into open threads in the field about the role of over-parameterization in learning. We also shed light on the existence of specific output dimensions which alone give very competitive results when compared to using all dimensions of output vectors.

arXiv comment7 pages

References in corpus (8)

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Learning to Generate Reviews and Discovering Sentiment
• How to Fine-Tune BERT for Text Classification?
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• Visualizing and Understanding the Effectiveness of BERT
• Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing