PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks

Abstract

Unsupervised text embedding methods, such as Skip-gram and Paragraph Vector, have been attracting increasing attention due to their simplicity, scalability, and effectiveness. However, comparing to sophisticated deep learning architectures such as convolutional neural networks, these methods usually yield inferior results when applied to particular machine learning tasks. One possible reason is that these text embedding methods learn the representation of text in a fully unsupervised way, without leveraging the labeled information available for the task. Although the low dimensional representations learned are applicable to many different tasks, they are not particularly tuned for any task. In this paper, we fill this gap by proposing a semi-supervised representation learning method for text data, which we call the \textit{predictive text embedding} (PTE). Predictive text embedding utilizes both labeled and unlabeled data to learn the embedding of text. The labeled information and different levels of word co-occurrence information are first represented as a large-scale heterogeneous text network, which is then embedded into a low dimensional space through a principled and efficient algorithm. This low dimensional embedding not only preserves the semantic closeness of words and documents, but also has a strong predictive power for the particular task. Compared to recent supervised approaches based on convolutional neural networks, predictive text embedding is comparable or more effective, much more efficient, and has fewer parameters to tune.

KDD 2015

References in corpus (2)

• DeepWalk: Online Learning of Social Representations
• LINE: Large-scale Information Network Embedding

Cited by in corpus (32)

• Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec
• Representation Learning for Attributed Multiplex Heterogeneous Network
• Graph Learning: A Survey
• A Comprehensive Survey on Deep Graph Representation Learning
• KST-GCN: A Knowledge-Driven Spatial-Temporal Graph Convolutional Network for Traffic Forecasting
• Weakly-Supervised Neural Text Classification
• Interpretable and Efficient Heterogeneous Graph Convolutional Network
• Easing Embedding Learning by Comprehensive Transcription of Heterogeneous Information Networks
• Hierarchically-Refined Label Attention Network for Sequence Labeling
• Representation Learning for Natural Language Processing
• Discriminative Topic Mining via Category-Name Guided Text Embedding
• Exploring Student Check-In Behavior for Improved Point-of-Interest Prediction
• TwHIN: Embedding the Twitter Heterogeneous Information Network for Personalized Recommendation
• Recommender systems based on graph embedding techniques: A comprehensive review
• AEGCN: An Autoencoder-Constrained Graph Convolutional Network
• Is a Single Vector Enough? Exploring Node Polysemy for Network Embedding
• Propositionalization and Embeddings: Two Sides of the Same Coin
• Embedding-based Silhouette Community Detection
• Beyond Low-Pass Filters: Adaptive Feature Propagation on Graphs
• PANE: scalable and effective attributed network embedding
• REFINE: Random RangE FInder for Network Embedding
• CoarSAS2hvec: Heterogeneous Information Network Embedding with Balanced Network Sampling
• Weakly Supervised Learning for Analyzing Political Campaigns on Facebook
• Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems
• Zoo Guide to Network Embedding
• SNoRe: Scalable Unsupervised Learning of Symbolic Node Representations
• Towards Deeper Understanding of PPR-based Embedding Approaches: A Topological Perspective
• Generating Post-hoc Explanations for Skip-gram-based Node Embeddings by Identifying Important Nodes with Bridgeness
• Symbolic Graph Embedding using Frequent Pattern Mining
• TANGNN: a Concise, Scalable and Effective Graph Neural Networks with Top-m Attention Mechanism for Graph Representation Learning
• ISHNE: Influence Self-attention for Heterogeneous Network Embedding
• ${\text{H}}^2\text{TNE}$: Temporal Heterogeneous Information Network Embedding in Hyperbolic Spaces