Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
arXiv:2109.04810
Abstract
Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. In this paper, we proposed Mixture-of-Partitions (MoP), an infusion approach that can handle a very large knowledge graph (KG) by partitioning it into smaller sub-graphs and infusing their specific knowledge into various BERT models using lightweight adapters. To leverage the overall factual knowledge for a target task, these sub-graph adapters are further fine-tuned along with the underlying BERT through a mixture layer. We evaluate our MoP with three biomedical BERTs (SciBERT, BioBERT, PubmedBERT) on six downstream tasks (inc. NLI, QA, Classification), and the results show that our MoP consistently enhances the underlying BERTs in task performance, and achieves new SOTA performances on five evaluated datasets.
arXiv commentEMNLP 2021 camera-ready version
References in corpus (11)
- Distributed Representations of Words and Phrases and their Compositionality
- Distilling the Knowledge in a Neural Network
- Decoupled Weight Decay Regularization
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Categorical Reparameterization with Gumbel-Softmax
- Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
- ERNIE: Enhanced Representation through Knowledge Integration
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
- K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
- DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs
- Improving Biomedical Pretrained Language Models with Knowledge