Improving Sentence Representations with Consensus Maximisation

Abstract

Consensus maximisation learning can provide self-supervision when different views are available of the same data. The distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present a new self-supervised learning framework for learning sentence representations which minimises the disagreement between two views of the same sentence where one view encodes the sentence with a recurrent neural network (RNN), and the other view encodes the same sentence with a simple linear model. After learning, the individual views (networks) result in higher quality sentence representations than their single-view learnt counterparts (learnt using only the distributional hypothesis) as judged by performance on standard downstream tasks. An ensemble of both views provides even better generalisation on both supervised and unsupervised downstream tasks. Also, importantly the ensemble of views trained with consensus maximisation between the two different architectures performs better on downstream tasks than an analogous ensemble made from the single-view trained counterparts.

arXiv commentarXiv admin note: substantial text overlap with arXiv:1805.07443

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Deep Residual Learning for Image Recognition
• From Frequency to Meaning: Vector Space Models of Semantics
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• A large annotated corpus for learning natural language inference
• An efficient framework for learning sentence representations
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Learned in Translation: Contextualized Word Vectors
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• DisSent: Sentence Representation Learning from Explicit Discourse Relations

Cited by in corpus (2)

• Learning Compressed Sentence Representations for On-Device Text Processing
• Text-to-Text Multi-view Learning for Passage Re-ranking