Visualizing and Understanding the Effectiveness of BERT
arXiv:1908.05620
Abstract
Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks. However, it is unclear why the pre-training-then-fine-tuning paradigm can improve performance and generalization capability across different tasks. In this paper, we propose to visualize loss landscapes and optimization trajectories of fine-tuning BERT on specific datasets. First, we find that pre-training reaches a good initial point across downstream tasks, which leads to wider optima and easier optimization compared with training from scratch. We also demonstrate that the fine-tuning procedure is robust to overfitting, even though BERT is highly over-parameterized for downstream tasks. Second, the visualization results indicate that fine-tuning BERT tends to generalize better because of the flat and wide optima, and the consistency between the training loss surface and the generalization error surface. Third, the lower layers of BERT are more invariant during fine-tuning, which suggests that the layers that are close to input learn more transferable representations of language.
arXiv commentAccepted by EMNLP-19
References in corpus (5)
Cited by in corpus (9)
- Perspectives of Non-Expert Users on Cyber Security and Privacy: An Analysis of Online Discussions on Twitter
- SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
- On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
- DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
- How Does Adversarial Fine-Tuning Benefit BERT?
- Emergent Properties of Finetuned Language Representation Models
- Multirate Training of Neural Networks
- Interpreting Language Models Through Knowledge Graph Extraction
- On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias