Semantic and Relational Spaces in Science of Science: Deep Learning Models for Article Vectorisation

Abstract

Over the last century, we observe a steady and exponentially growth of scientific publications globally. The overwhelming amount of available literature makes a holistic analysis of the research within a field and between fields based on manual inspection impossible. Automatic techniques to support the process of literature review are required to find the epistemic and social patterns that are embedded in scientific publications. In computer sciences, new tools have been developed to deal with large volumes of data. In particular, deep learning techniques open the possibility of automated end-to-end models to project observations to a new, low-dimensional space where the most relevant information of each observation is highlighted. Using deep learning to build new representations of scientific publications is a growing but still emerging field of research. The aim of this paper is to discuss the potential and limits of deep learning for gathering insights about scientific research articles. We focus on document-level embeddings based on the semantic and relational aspects of articles, using Natural Language Processing (NLP) and Graph Neural Networks (GNNs). We explore the different outcomes generated by those techniques. Our results show that using NLP we can encode a semantic space of articles, while with GNN we are able to build a relational space where the social practices of a research community are also encoded.

References in corpus (22)

• Scikit-learn: Machine Learning in Python
• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Semantics derived automatically from language corpora contain human-like biases
• Spectral Networks and Locally Connected Networks on Graphs
• Deep Learning using Rectified Linear Units (ReLU)
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Graph Neural Networks: A Review of Methods and Applications
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• Fast Graph Representation Learning with PyTorch Geometric
• Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes
• A correlated topic model of Science
• Variational Graph Auto-Encoders
• The Geometry of Culture: Analyzing Meaning through Word Embeddings
• A Gentle Introduction to Deep Learning for Graphs
• Representation Learning on Graphs: Methods and Applications
• Link Prediction Based on Graph Neural Networks
• Attention-based Graph Neural Network for Semi-supervised Learning
• Exploiting citation networks for large-scale author name disambiguation

Cited by in corpus (3)

• Citation proximus: the role of social and semantic ties in citing behaviour
• Graph Embedding for Mapping Interdisciplinary Research Networks
• Citation importance-aware document representation learning for large-scale science mapping