Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification

Abstract

While paragraph embedding models are remarkably effective for downstream classification tasks, what they learn and encode into a single vector remains opaque. In this paper, we investigate a state-of-the-art paragraph embedding method proposed by Zhang et al. (2017) and discover that it cannot reliably tell whether a given sentence occurs in the input paragraph or not. We formulate a sentence content task to probe for this basic linguistic property and find that even a much simpler bag-of-words method has no trouble solving it. This result motivates us to replace the reconstruction-based objective of Zhang et al. (2017) with our sentence content probe objective in a semi-supervised setting. Despite its simplicity, our objective improves over paragraph reconstruction in terms of (1) downstream classification accuracies on benchmark datasets, (2) faster training, and (3) better generalization ability.

arXiv commentAccepted as a conference paper at ACL 2019

References in corpus (10)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Distributed Representations of Sentences and Documents
• Universal Sentence Encoder
• Skip-Thought Vectors
• What do Neural Machine Translation Models Learn about Morphology?
• Document Embedding with Paragraph Vectors
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• DisSent: Sentence Representation Learning from Explicit Discourse Relations