Hierarchical Neural Language Models for Joint Representation of Streaming Documents and their Content

Abstract

We consider the problem of learning distributed representations for documents in data streams. The documents are represented as low-dimensional vectors and are jointly learned with distributed vector representations of word tokens using a hierarchical framework with two embedded neural language models. In particular, we exploit the context of documents in streams and use one of the language models to model the document sequences, and the other to model word sequences within them. The models learn continuous vector representations for both word tokens and documents such that semantically similar documents and words are close in a common vector space. We discuss extensions to our model, which can be applied to personalized recommendation and social relationship mining by adding further user layers to the hierarchy, thus learning user-specific vectors to represent individual preferences. We validated the learned representations on a public movie rating data set from MovieLens, as well as on a large-scale Yahoo News data comprising three months of user activity logs collected on Yahoo servers. The results indicate that the proposed model can learn useful representations of both documents and word tokens, outperforming the current state-of-the-art by a large margin.

arXiv comment24th International World Wide Web Conference

References in corpus (6)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• DeepWalk: Online Learning of Social Representations
• Distributed Representations of Sentences and Documents
• A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
• A Multiplicative Model for Learning Distributed Text-Based Attribute Representations

Cited by in corpus (11)

• Meta-Prod2Vec - Product Embeddings Using Side-Information for Recommendation
• Regularizing Matrix Factorization with User and Item Embeddings for Recommendation
• Network Representation Learning: A Survey
• Simultaneous Learning of Trees and Representations for Extreme Classification and Density Estimation
• Inferring Complementary Products from Baskets and Browsing Sessions
• ParVecMF: A Paragraph Vector-based Matrix Factorization Recommender System
• CB2CF: A Neural Multiview Content-to-Collaborative Filtering Model for Completely Cold Item Recommendations
• Goal-driven Command Recommendations for Analysts
• Getting Started with Neural Models for Semantic Matching in Web Search
• KeyVec: Key-semantics Preserving Document Representations
• Improving Tweet Representations using Temporal and User Context