Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning

Abstract

The encoder-decoder models for unsupervised sentence representation learning tend to discard the decoder after being trained on a large unlabelled corpus, since only the encoder is needed to map the input sentence into a vector representation. However, parameters learnt in the decoder also contain useful information about language. In order to utilise the decoder after learning, we present two types of decoding functions whose inverse can be easily derived without expensive inverse calculation. Therefore, the inverse of the decoding function serves as another encoder that produces sentence representations. We show that, with careful design of the decoding functions, the model learns good sentence representations, and the ensemble of the representations produced from the encoder and the inverse of the decoder demonstrate even better generalisation ability and solid transferability.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• On the difficulty of training Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Skip-Thought Vectors
• An efficient framework for learning sentence representations
• Improving Variational Inference with Inverse Autoregressive Flow
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• No Training Required: Exploring Random Encoders for Sentence Classification
• DisSent: Sentence Representation Learning from Explicit Discourse Relations