Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion
arXiv:2401.12997 · doi:10.1609/aaai.v38i8.28680
Abstract
In recent years, knowledge graph completion (KGC) models based on pre-trained language model (PLM) have shown promising results. However, the large number of parameters and high computational cost of PLM models pose challenges for their application in downstream tasks. This paper proposes a progressive distillation method based on masked generation features for KGC task, aiming to significantly reduce the complexity of pre-trained models. Specifically, we perform pre-distillation on PLM to obtain high-quality teacher models, and compress the PLM network to obtain multi-grade student models. However, traditional feature distillation suffers from the limitation of having a single representation of information in teacher models. To solve this problem, we propose masked generation of teacher-student features, which contain richer representation information. Furthermore, there is a significant gap in representation ability between teacher and student. Therefore, we design a progressive distillation method to distill student models at each grade level, enabling efficient knowledge transfer from teachers to students. The experimental results demonstrate that the model in the pre-distillation stage surpasses the existing state-of-the-art methods. Furthermore, in the progressive distillation stage, the model significantly reduces the model parameters while maintaining a certain level of performance. Specifically, the model parameters of the lower-grade student model are reduced by 56.7\% compared to the baseline.
Accepted by AAAI2024
References in corpus (15)
- Distilling the Knowledge in a Neural Network
- Complex Embeddings for Simple Link Prediction
- A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network
- RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
- Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences
- KG-BERT: BERT for Knowledge Graph Completion
- Composition-based Multi-Relational Graph Convolutional Networks
- End-to-end Structure-Aware Convolutional Networks for Knowledge Base Completion
- Focal and Global Knowledge Distillation for Detectors
- Revisiting Self-Distillation
- DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoning
- GreenKGC: A Lightweight Knowledge Graph Completion Method
- Language Models as Knowledge Embeddings
- On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
- Learning From Yourself: A Self-Distillation Method for Fake Speech Detection