GiBERT: Introducing Linguistic Knowledge into BERT through a Lightweight Gated Injection Method
arXiv:2010.12532
Abstract
Large pre-trained language models such as BERT have been the driving force behind recent improvements across many NLP tasks. However, BERT is only trained to predict missing words - either behind masks or in the next sentence - and has no knowledge of lexical, syntactic or semantic information beyond what it picks up through unsupervised pre-training. We propose a novel method to explicitly inject linguistic knowledge in the form of word embeddings into any layer of a pre-trained BERT. Our performance improvements on multiple semantic similarity datasets when injecting dependency-based and counter-fitted embeddings indicate that such information is beneficial and currently missing from the original model. Our qualitative analysis shows that counter-fitted embedding injection particularly helps with cases involving synonym pairs.
References in corpus (16)
- Efficient Estimation of Word Representations in Vector Space
- Natural Language Inference over Interaction Space
- Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
- Text Matching as Image Recognition
- K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
- Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
- InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
- AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
- Knowledge Distillation from Internal Representations
- A BERT based Sentiment Analysis and Key Entity Detection Approach for Online Financial Texts
- Fine-Tuning BERT for Schema-Guided Zero-Shot Dialogue State Tracking
- Few-Shot Self Reminder to Overcome Catastrophic Forgetting
- Knowledge Enhanced Contextual Word Representations
- FGN: Fusion Glyph Network for Chinese Named Entity Recognition
- BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
- A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems