K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Abstract

We study the problem of injecting knowledge into large pre-trained models like BERT and RoBERTa. Existing methods typically update the original parameters of pre-trained models when injecting knowledge. However, when multiple kinds of knowledge are injected, the historically injected knowledge would be flushed away. To address this, we propose K-Adapter, a framework that retains the original parameters of the pre-trained model fixed and supports the development of versatile knowledge-infused model. Taking RoBERTa as the backbone model, K-Adapter has a neural adapter for each kind of infused knowledge, like a plug-in connected to RoBERTa. There is no information flow between different adapters, thus multiple adapters can be efficiently trained in a distributed way. As a case study, we inject two kinds of knowledge in this work, including (1) factual knowledge obtained from automatically aligned text-triplets on Wikipedia and Wikidata and (2) linguistic knowledge obtained via dependency parsing. Results on three knowledge-driven tasks, including relation classification, entity typing, and question answering, demonstrate that each adapter improves the performance and the combination of both adapters brings further improvements. Further analysis indicates that K-Adapter captures versatile knowledge than RoBERTa.

References in corpus (10)

• Overcoming catastrophic forgetting in neural networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Bidirectional Attention Flow for Machine Comprehension
• Learning multiple visual domains with residual adapters
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering
• Quasar: Datasets for Question Answering by Search and Reading
• Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
• R${}^3$: Reinforced Reader-Ranker for Open-Domain Question Answering
• Integrating Graph Contextualized Knowledge into Pre-trained Language Models

Cited by in corpus (47)

• Pre-trained Models for Natural Language Processing: A Survey
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• An Improved Baseline for Sentence-level Relation Extraction
• Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• Multimodal Parameter-Efficient Few-Shot Class Incremental Learning
• LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• A Survey of Knowledge Enhanced Pre-trained Models
• CoLAKE: Contextualized Language and Knowledge Embedding
• E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce
• KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs
• Language Models As or For Knowledge Bases
• Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs
• One Person, One Model, One World: Learning Continual User Representation without Forgetting
• Pre-Trained Models: Past, Present and Future
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
• Pre-training Text-to-Text Transformers for Concept-centric Common Sense
• CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision
• Combining pre-trained language models and structured knowledge
• KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning
• Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter
• Unsupervised Domain Adaptation with Adapter
• KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding
• Does Knowledge Help General NLU? An Empirical Study
• ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration
• Adversarial Training for Commonsense Inference
• GAN Memory with No Forgetting
• Relation Classification with Entity Type Restriction
• AIO-P: Expanding Neural Performance Predictors Beyond Image Classification
• Knowledge Transfer via Pre-training for Recommendation: A Review and Prospect
• Billion-scale Pre-trained E-commerce Product Knowledge Graph Model
• SYNERGY: Building Task Bots at Scale Using Symbolic Knowledge and Machine Teaching
• Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
• Syntax-Enhanced Pre-trained Model
• BERTnesia: Investigating the capture and forgetting of knowledge in BERT
• Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems
• Focusing Knowledge-based Graph Argument Mining via Topic Modeling
• K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce
• Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction
• Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers
• THEaiTRE 1.0: Interactive generation of theatre play scripts
• K-AID: Enhancing Pre-trained Language Models with Domain Knowledge for Question Answering
• GiBERT: Introducing Linguistic Knowledge into BERT through a Lightweight Gated Injection Method
• Metadata Shaping: Natural Language Annotations for the Tail
• Interpretable Entity Representations through Large-Scale Typing