Augmenting semantic lexicons using word embeddings and transfer learning
arXiv:2109.09010 · doi:10.3389/frai.2021.783778
Abstract
Sentiment-aware intelligent systems are essential to a wide array of applications. These systems are driven by language models which broadly fall into two paradigms: Lexicon-based and contextual. Although recent contextual models are increasingly dominant, we still see demand for lexicon-based models because of their interpretability and ease of use. For example, lexicon-based models allow researchers to readily determine which words and phrases contribute most to a change in measured sentiment. A challenge for any lexicon-based approach is that the lexicon needs to be routinely expanded with new words and expressions. Here, we propose two models for automatic lexicon expansion. Our first model establishes a baseline employing a simple and shallow neural network initialized with pre-trained word embeddings using a non-contextual approach. Our second model improves upon our baseline, featuring a deep Transformer-based network that brings to bear word definitions to estimate their lexical polarity. Our evaluation shows that both models are able to score new words with a similar accuracy to reviewers from Amazon Mechanical Turk, but at a fraction of the cost.
arXiv comment17 pages, 8 figures
References in corpus (18)
- Efficient Estimation of Word Representations in Vector Space
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distributed Representations of Words and Phrases and their Compositionality
- Distilling the Knowledge in a Neural Network
- TensorFlow: A system for large-scale machine learning
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- Enhanced LSTM for Natural Language Inference
- Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter
- The Computational Limits of Deep Learning
- Measuring the happiness of large-scale written expression: Songs, Blogs, and Presidents
- Get out the vote: Determining support or opposition from Congressional floor-debate transcripts
- Generalized Word Shift Graphs: A Method for Visualizing and Explaining Pairwise Comparisons Between Texts
- How the world's collective attention is being paid to a pandemic: COVID-19 related n-gram time series for 24 languages on Twitter
- Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing
- Predicting Concreteness and Imageability of Words Within and Across Languages via Word Embeddings
- Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter
- Quantifying language changes surrounding mental health on Twitter