Text analysis and deep learning: A network approach
arXiv:2110.04151
Abstract
Much information available to applied researchers is contained within written language or spoken text. Deep language models such as BERT have achieved unprecedented success in many applications of computational linguistics. However, much less is known about how these models can be used to analyze existing text. We propose a novel method that combines transformer models with network analysis to form a self-referential representation of language use within a corpus of interest. Our approach produces linguistic relations strongly consistent with the underlying model as well as mathematically well-defined operations on them, while reducing the amount of discretionary choices of representation and distance measures. It represents, to the best of our knowledge, the first unsupervised method to extract semantic networks directly from deep language models. We illustrate our approach in a semantic analysis of the term "founder". Using the entire corpus of Harvard Business Review from 1980 to 2020, we find that ties in our network track the semantics of discourse over time, and across contexts, identifying and relating clusters of semantic and syntactic relations. Finally, we discuss how this method can also complement and inform analyses of the behavior of deep learning models.
References in corpus (12)
- Efficient Estimation of Word Representations in Vector Space
- Fast unfolding of communities in large networks
- From Louvain to Leiden: guaranteeing well-connected communities
- Maps of random walks on complex networks reveal community structure
- Language Models are Few-Shot Learners
- Whitening Sentence Representations for Better Semantics and Faster Retrieval
- Analysing Lexical Semantic Change with Contextualised Word Representations
- Survey of Computational Approaches to Lexical Semantic Change
- Very Deep Transformers for Neural Machine Translation
- Attention Interpretability Across NLP Tasks
- Heated-Up Softmax Embedding
- OpenHowNet: An Open Sememe-based Lexical Knowledge Base