Trimming and Improving Skip-thought Vectors

Abstract

The skip-thought model has been proven to be effective at learning sentence representations and capturing sentence semantics. In this paper, we propose a suite of techniques to trim and improve it. First, we validate a hypothesis that, given a current sentence, inferring the previous and inferring the next sentence provide similar supervision power, therefore only one decoder for predicting the next sentence is preserved in our trimmed skip-thought model. Second, we present a connection layer between encoder and decoder to help the model to generalize better on semantic relatedness tasks. Third, we found that a good word embedding initialization is also essential for learning better sentence representations. We train our model unsupervised on a large corpus with contiguous sentences, and then evaluate the trained model on 7 supervised tasks, which includes semantic relatedness, paraphrase detection, and text classification benchmarks. We empirically show that, our proposed model is a faster, lighter-weight and equally powerful alternative to the original skip-thought model.

References in corpus (6)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Enhanced LSTM for Natural Language Inference
• Semi-supervised Sequence Learning
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

Cited by in corpus (1)

• An end-to-end Neural Network Framework for Text Clustering