P-SIF: Document Embeddings Using Partition Averaging
arXiv:2005.09069
Abstract
Simple weighted averaging of word vectors often yields effective representations for sentences which outperform sophisticated seq2seq neural models in many tasks. While it is desirable to use the same method to represent documents as well, unfortunately, the effectiveness is lost when representing long documents involving multiple sentences. One of the key reasons is that a longer document is likely to contain words from many different topics; hence, creating a single vector while ignoring all the topical structure is unlikely to yield an effective document representation. This problem is less acute in single sentences and other short text fragments where the presence of a single topic is most likely. To alleviate this problem, we present P-SIF, a partitioned word averaging model to represent long documents. P-SIF retains the simplicity of simple weighted word averaging while taking a document's topical structure into account. In particular, P-SIF learns topic-specific vectors from a document and finally concatenates them all to represent the overall document. We provide theoretical justifications on the correctness of P-SIF. Through a comprehensive set of experiments, we demonstrate P-SIF's effectiveness compared to simple weighted averaging and many other baselines.
arXiv comment15 Pages, 3 Figures, 13 Tables, AAAI 2020, Blog : http://vivgupt.blogspot.com/2019/06/document-vector-estimation-using.html
References in corpus (11)
- Distributed Representations of Words and Phrases and their Compositionality
- Distributed Representations of Sentences and Documents
- Universal Sentence Encoder
- Skip-Thought Vectors
- Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
- SentEval: An Evaluation Toolkit for Universal Sentence Representations
- Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec
- Evaluation of sentence embeddings in downstream and linguistic probing tasks
- From Paraphrase Database to Compositional Paraphrase Model and Back
- The Dynamic Embedded Topic Model
- LTSG: Latent Topical Skip-Gram for Mutually Learning Topic Model and Vector Representations