RPT: Toward Transferable Model on Heterogeneous Researcher Data via Pre-Training
arXiv:2110.07336 · doi:10.1109/TBDATA.2022.3152386
Abstract
With the growth of the academic engines, the mining and analysis acquisition of massive researcher data, such as collaborator recommendation and researcher retrieval, has become indispensable. It can improve the quality of services and intelligence of academic engines. Most of the existing studies for researcher data mining focus on a single task for a particular application scenario and learning a task-specific model, which is usually unable to transfer to out-of-scope tasks. The pre-training technology provides a generalized and sharing model to capture valuable information from enormous unlabeled data. The model can accomplish multiple downstream tasks via a few fine-tuning steps. In this paper, we propose a multi-task self-supervised learning-based researcher data pre-training model named RPT. Specifically, we divide the researchers' data into semantic document sets and community graph. We design the hierarchical Transformer and the local community encoder to capture information from the two categories of data, respectively. Then, we propose three self-supervised learning objectives to train the whole model. Finally, we also propose two transfer modes of RPT for fine-tuning in different scenarios. We conduct extensive experiments to evaluate RPT, results on three downstream tasks verify the effectiveness of pre-training for researcher data mining.
arXiv commentAccepted as regular paper by IEEE Transactions on Big Data
References in corpus (22)
- Efficient Estimation of Word Representations in Vector Space
- Distributed Representations of Words and Phrases and their Compositionality
- DeepWalk: Online Learning of Social Representations
- Semi-Supervised Classification with Graph Convolutional Networks
- A Simple Framework for Contrastive Learning of Visual Representations
- Representation Learning with Contrastive Predictive Coding
- Distributed Representations of Sentences and Documents
- LINE: Large-scale Information Network Embedding
- Unsupervised Representation Learning by Predicting Image Rotations
- Graph Contrastive Learning with Augmentations
- Hierarchical Graph Representation Learning with Differentiable Pooling
- GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training
- word2vec Parameter Learning Explained
- DeepInf: Social Influence Prediction with Deep Learning
- Attributed Social Network Embedding
- Rethinking Pre-training and Self-training
- GraphSAINT: Graph Sampling Based Inductive Learning Method
- Scientific Article Recommendation: Exploiting Common Author Relations and Historical Preferences
- Can Scientific Impact Be Predicted?
- GPT-GNN: Generative Pre-Training of Graph Neural Networks
- Scalable Graph Neural Networks via Bidirectional Propagation
- Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning