From Frequency to Meaning: Vector Space Models of Semantics

Abstract

Computers understand very little of the meaning of human language. This profoundly limits our ability to give instructions to computers, the ability of computers to explain their actions to us, and the ability of computers to analyse and process text. Vector space models (VSMs) of semantics are beginning to address these limits. This paper surveys the use of VSMs for semantic processing of text. We organize the literature on VSMs according to the structure of the matrix in a VSM. There are currently three broad classes of VSMs, based on term-document, word-context, and pair-pattern matrices, yielding three classes of applications. We survey a broad range of applications in these three categories and we take a detailed look at a specific open source project in each category. Our goal in this survey is to show the breadth of applications of VSMs for semantics, to provide a new perspective on VSMs for those who are already familiar with the area, and to provide pointers into the literature for those who are less familiar with the field.

References in corpus (14)

• Machine Learning in Automated Text Categorization
• Empirical Analysis of Predictive Algorithms for Collaborative Filtering
• Using Information Content to Evaluate Semantic Similarity in a Taxonomy
• Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy
• Thumbs up? Sentiment Classification using Machine Learning Techniques
• Advances in domain independent linear text segmentation
• Measuring Praise and Criticism: Inference of Semantic Orientation from Association
• Measuring Semantic Similarity by Latent Relational Analysis
• Combining Independent Modules to Solve Multiple-choice Synonym and Analogy Problems
• Discrete Component Analysis
• Similarity-Based Models of Word Cooccurrence Probabilities
• Empirical Evaluation of Four Tensor Decomposition Algorithms
• A Uniform Approach to Analogies, Synonyms, Antonyms, and Associations
• Corpus-based Learning of Analogies and Semantic Relations

Cited by in corpus (221)

• Distributed Representations of Words and Phrases and their Compositionality
• Distributed Representations of Sentences and Documents
• Semantics derived automatically from language corpora contain human-like biases
• Recent Trends in Deep Learning Based Natural Language Processing
• Zero-Shot Learning Through Cross-Modal Transfer
• Enriching Word Vectors with Subword Information
• Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
• Measuring academic influence: Not all citations are equal
• Distributional Semantics and Linguistic Theory
• Recurrent Convolutional Neural Networks for Discourse Compositionality
• A Survey of Paraphrasing and Textual Entailment Methods
• Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change
• Exploring Chemical Space using Natural Language Processing Methodologies for Drug Discovery
• Semantic Similarity from Natural Language and Ontology Analysis
• Diachronic word embeddings and semantic shifts: a survey
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• On the Dimensionality of Word Embedding
• Prediction of miRNA-disease associations with a vector space model
• Neural Models for Information Retrieval
• An efficient streaming algorithm for spectral proper orthogonal decomposition
• Comparative study of LSA vs Word2vec embeddings in small corpora: a case study in dreams database
• A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions
• Jointly Learning Word Embeddings and Latent Topics
• Multilingual Distributed Representations without Word Alignment
• Domain and Function: A Dual-Space Model of Semantic Relations and Compositions
• Separated by an Un-common Language: Towards Judgment Language Informed Vector Space Modeling
• Word Emdeddings through Hellinger PCA
• Semisupervised Autoencoder for Sentiment Analysis
• Two Step CCA: A new spectral method for estimating vector models of words
• Multi-sense embeddings through a word sense disambiguation process
• Embedding Word Similarity with Neural Machine Translation
• Joint Word Representation Learning using a Corpus and a Semantic Lexicon
• Not All Neural Embeddings are Born Equal
• Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora
• SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation
• Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change
• Revisiting Summarization Evaluation for Scientific Articles
• Thinking, Fast and Slow: Combining Vector Spaces and Knowledge Graphs
• Corpus-Based Paraphrase Detection Experiments and Review
• Lexically-Accelerated Dense Retrieval
• Category-Theoretic Quantitative Compositional Distributional Models of Natural Language Semantics
• Texts in, meaning out: neural language models in semantic similarity task for Russian
• Word Embeddings: A Survey
• Augur: Mining Human Behaviors from Fiction to Power Interactive Systems
• Topic2Vec: Learning Distributed Representations of Topics
• Symbolic, Distributed and Distributional Representations for Natural Language Processing in the Era of Deep Learning: a Survey
• Centrality-as-Relevance: Support Sets and Similarity as Geometric Proximity
• Semantic projection: recovering human knowledge of multiple, distinct object features from word embeddings
• Automated Coding of Under-Studied Medical Concept Domains: Linking Physical Activity Reports to the International Classification of Functioning, Disability, and Health
• Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: The Impact of Word Representation on Sequence Labelling Tasks
• Experiments with Three Approaches to Recognizing Lexical Entailment
• Quantum Mathematics in Artificial Intelligence
• Learning Chinese Word Representations From Glyphs Of Characters
• Advances in Machine Learning for the Behavioral Sciences
• Contextual Word Representations: A Contextual Introduction
• VISALOGY: Answering Visual Analogy Questions
• Effectiveness of Data-Driven Induction of Semantic Spaces and Traditional Classifiers for Sarcasm Detection
• Machine Learning Techniques with Ontology for Subjective Answer Evaluation
• Multiplex Communities and the Emergence of International Conflict
• Medical Synonym Extraction with Concept Space Models
• BERT has a Moral Compass: Improvements of ethical and moral values of machines
• Compositionality decomposed: how do neural networks generalise?
• Evaluating Neural Word Representations in Tensor-Based Compositional Settings
• One Vector is Not Enough: Entity-Augmented Distributional Semantics for Discourse Relations
• Distributional semantic modeling: a revised technique to train term/word vector space models applying the ontology-related approach
• Extracting Food Substitutes From Food Diary via Distributional Similarity
• Embedding Semantic Relations into Word Representations
• Modeling Musical Context with Word2vec
• Entity Embeddings with Conceptual Subspaces as a Basis for Plausible Reasoning
• Watset: Local-Global Graph Clustering with Applications in Sense and Frame Induction
• Topical alignment in online social systems
• Distributional Modeling on a Diet: One-shot Word Learning from Text Only
• CogALex-V Shared Task: LexNET - Integrated Path-based and Distributional Method for the Identification of Semantic Relations
• Incremental Skip-gram Model with Negative Sampling
• LTM: Scalable and Black-box Similarity-based Test Suite Minimization based on Language Models
• Revisiting the Context Window for Cross-lingual Word Embeddings
• Vector representations of text data in deep learning
• On the Semantic Interpretability of Artificial Intelligence Models
• Sparse Overcomplete Word Vector Representations
• Leverage Financial News to Predict Stock Price Movements Using Word Embeddings and Deep Neural Networks
• An Analysis on the Learning Rules of the Skip-Gram Model
• A Rank-Based Similarity Metric for Word Embeddings
• Lacking the embedding of a word? Look it up into a traditional dictionary
• A Measure of Similarity in Textual Data Using Spearman's Rank Correlation Coefficient
• Characterizing Diseases from Unstructured Text: A Vocabulary Driven Word2vec Approach
• Semantic Composition and Decomposition: From Recognition to Generation
• The Mechanism of Additive Composition
• Distributional Inclusion Hypothesis for Tensor-based Composition
• Incremental dimension reduction of tensors with random index
• A framework for anomaly detection using language modeling, and its applications to finance
• LexBoost: Improving Lexical Document Retrieval with Nearest Neighbors
• Graph powering and spectral robustness
• An Empirical Study on Post-processing Methods for Word Embeddings
• KNET: A General Framework for Learning Word Embedding using Morphological Knowledge
• Word and Document Embeddings based on Neural Network Approaches
• Semantic Word Clusters Using Signed Normalized Graph Cuts
• Deconstructing and reconstructing word embedding algorithms
• Interpreting Knowledge Graph Relation Representation from Word Embeddings
• Group-theoretical vector space model
• Evaluation of taxonomic and neural embedding methods for calculating semantic similarity
• Non-distributive logics: from semantics to meaning
• On Event Causality Detection in Tweets
• Sentiment Analysis on IMDB Movie Comments and Twitter Data by Machine Learning and Vector Space Techniques
• Determining the Characteristic Vocabulary for a Specialized Dictionary using Word2vec and a Directed Crawler
• Using Machine Learning to Detect Fraudulent SMSs in Chichewa
• "The Sum of Its Parts": Joint Learning of Word and Phrase Representations with Autoencoders
• Semantic Composition via Probabilistic Model Theory
• Composing and Embedding the Words-as-Classifiers Model of Grounded Semantics
• Sentiment Analysis with Contextual Embeddings and Self-Attention
• Two SVDs produce more focal deep learning representations
• A Survey On Neural Word Embeddings
• Multi-view Sentence Representation Learning
• SeVeN: Augmenting Word Embeddings with Unsupervised Relation Vectors
• Features of word similarity
• Monitoring Term Drift Based on Semantic Consistency in an Evolving Vector Field
• Testing APSyn against Vector Cosine on Similarity Estimation
• Real-Time Steganalysis for Stream Media Based on Multi-channel Convolutional Sliding Windows
• Representing Meaning with a Combination of Logical and Distributional Models
• Investigating the Effect of Segmentation Methods on Neural Model based Sentiment Analysis on Informal Short Texts in Turkish
• Words, Concepts, and the Geometry of Analogy
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models
• Dirichlet-Smoothed Word Embeddings for Low-Resource Settings
• A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains
• Learning Semantic Relatedness From Human Feedback Using Metric Learning
• Supervised Fine Tuning for Word Embedding with Integrated Knowledge
• Size vs. Structure in Training Corpora for Word Embedding Models: Araneum Russicum Maximum and Russian National Corpus
• Second-Order Word Embeddings from Nearest Neighbor Topological Features
• Combating the Filter Bubble: Designing for Serendipity in a University Course Recommendation System
• Identification, Interpretability, and Bayesian Word Embeddings
• Towards Uncertainty Aware Task Delegation and Human-AI Collaborative Decision-Making
• Computational Hermeneutics: Evaluating generative AI as a cultural technology
• Traj2User: exploiting embeddings for computing similarity of users mobile behavior
• Textual Membership Queries
• Learning Semantically and Additively Compositional Distributional Representations
• Words are not Equal: Graded Weighting Model for building Composite Document Vectors
• Towards a Better Understanding of Predict and Count Models
• Probabilistic Modelling of Morphologically Rich Languages
• Learning Type-Driven Tensor-Based Meaning Representations
• Conceptual Text Summarizer: A new model in continuous vector space
• Integrating Dictionary Feature into A Deep Learning Model for Disease Named Entity Recognition
• Linguists Who Use Probabilistic Models Love Them: Quantification in Functional Distributional Semantics
• A survey on extremism analysis using Natural Language Processing
• Discovering Hypernymy in Text-Rich Heterogeneous Information Network by Exploiting Context Granularity
• Getting Started with Neural Models for Semantic Matching in Web Search
• Identifying Metaphoric Antonyms in a Corpus Analysis of Finance Articles
• Improving Sentence Representations with Consensus Maximisation
• Drivers of the decrease of patent similarities from 1976 to 2021
• Feature extraction using Latent Dirichlet Allocation and Neural Networks: A case study on movie synopses
• Factorial Hidden Markov Models for Learning Representations of Natural Language
• On the selection of the correct number of terms for profile construction: theoretical and empirical analysis
• Antonym-Synonym Classification Based on New Sub-space Embeddings
• A Physical Metaphor to Study Semantic Drift
• A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments
• Unsupervised Measure of Word Similarity: How to Outperform Co-occurrence and Vector Cosine in VSMs
• Data-Driven Application Maintenance: Views from the Trenches
• SimDoc: Topic Sequence Alignment based Document Similarity Framework
• Attention improves concentration when learning node embeddings
• Distributed Vector Representations of Folksong Motifs
• Extending and Improving Wordnet via Unsupervised Word Embeddings
• Fast Extraction of Word Embedding from Q-contexts
• Classical Copying versus Quantum Entanglement in Natural Language: The Case of VP-ellipsis
• Mental representations of objects reflect the ways in which we interact with them
• RiverText: A Python Library for Training and Evaluating Incremental Word Embeddings from Text Data Streams
• Relation Extraction Datasets in the Digital Humanities Domain and their Evaluation with Word Embeddings
• Cosine Similarity of Multimodal Content Vectors for TV Programmes
• Word2vec Conjecture and A Limitative Result
• Semantic Analysis for Automated Evaluation of the Potential Impact of Research Articles
• A case for deep learning in semantics
• Neural Latent Relational Analysis to Capture Lexical Semantic Relations in a Vector Space
• Characterizing Departures from Linearity in Word Translation
• Joint Semantic Synthesis and Morphological Analysis of the Derived Word
• DialPort: Connecting the Spoken Dialog Research Community to Real User Data
• Multiplex Word Embeddings for Selectional Preference Acquisition
• Learning Taxonomies of Concepts and not Words using Contextualized Word Representations: A Position Paper
• Representation Learning for Words and Entities
• A New Approach for Measuring Sentiment Orientation based on Multi-Dimensional Vector Space
• Paraphrase-Supervised Models of Compositionality
• Genetic Optimization of Keywords Subset in the Classification Analysis of Texts Authorship
• A Latent Variable Model Approach to PMI-based Word Embeddings
• Distinguishing Antonyms and Synonyms in a Pattern-based Neural Network
• Radical-Enhanced Chinese Character Embedding
• Creation and Evaluation of Datasets for Distributional Semantics Tasks in the Digital Humanities Domain
• Diachronic Text Mining Investigation of Therapeutic Candidates for COVID-19
• CogniFNN: A Fuzzy Neural Network Framework for Cognitive Word Embedding Evaluation
• Identifying Metaphor Hierarchies in a Corpus Analysis of Finance Articles
• Sketching Linear Classifiers over Data Streams
• affinity: A System for Latent User Similarity Comparison on Texting Data
• Feature Learning for Meta-Paths in Knowledge Graphs
• A Simple and Efficient Probabilistic Language model for Code-Mixed Text
• Realizing RCC8 networks using convex regions
• R-grams: Unsupervised Learning of Semantic Units in Natural Language
• Learning Word Embeddings with Domain Awareness
• Political Footprints: Political Discourse Analysis using Pre-Trained Word Vectors
• A WL-SPPIM Semantic Model for Document Classification
• Compositional Approaches for Representing Relations Between Words: A Comparative Study
• Measuring Issue Ownership using Word Embeddings
• Tiered Clustering to Improve Lexical Entailment
• Reasoning-Driven Question-Answering for Natural Language Understanding
• Identifying and Explaining Discriminative Attributes
• Network representation learning: A macro and micro view
• Grover's Algorithm for Question Answering
• Ideological Consumerism in Colombian Elections, 2015: Links between Political Ideology, Twitter Activity and Electoral Results
• Towards Using Diachronic Distributed Word Representations as Models of Lexical Development
• Compositional Distributional Cognition
• A Factorized Model for Transitive Verbs in Compositional Distributional Semantics
• Semantic Modeling of Analytic-based Relationships with Direct Qualification
• Effective Combination of Language and Vision Through Model Composition and the R-CCA Method
• TEDL: A Text Encryption Method Based on Deep Learning
• On Quantum Generalizations of Information-Theoretic Measures and their Contribution to Distributional Semantics
• SEMIE: SEMantically Infused Embeddings with Enhanced Interpretability for Domain-specific Small Corpus
• A Concern Analysis of FOMC Statements Comparing The Great Recession and The COVID-19 Pandemic
• Image Annotation using Multi-Layer Sparse Coding
• Tracing metaphors in time through self-distance in vector spaces
• Blind signal decomposition of various word embeddings based on join and individual variance explained
• Logic, Language, and Calculus
• Scalable Methods for Calculating Term Co-Occurrence Frequencies
• An Improved Historical Embedding without Alignment
• Diversifying Topic-Coherent Response Generation for Natural Multi-turn Conversations
• "Show me the cup": Reference with Continuous Representations
• Entity-Augmented Distributional Semantics for Discourse Relations
• Bilingual Dictionary Induction for Bantu Languages