Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

Abstract

This work presents a novel objective function for the unsupervised training of neural network sentence encoders. It exploits signals from paragraph-level discourse coherence to train these models to understand text. Our objective is purely discriminative, allowing us to train models many times faster than was possible under prior methods, and it yields models which perform well in extrinsic evaluations.

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Embracing data abundance: BookTest Dataset for Reading Comprehension

Cited by in corpus (31)

• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• An efficient framework for learning sentence representations
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Correlating neural and symbolic representations of language
• DisSent: Sentence Representation Learning from Explicit Discourse Relations
• Learning Compressed Sentence Representations for On-Device Text Processing
• Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
• Using previous acoustic context to improve Text-to-Speech synthesis
• Pre-training Text Representations as Meta Learning
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• Trimming and Improving Skip-thought Vectors
• Multi-Label Transfer Learning for Multi-Relational Semantic Similarity
• Which *BERT? A Survey Organizing Contextualized Encoders
• Neural Skill Transfer from Supervised Language Tasks to Reading Comprehension
• Multi-view Sentence Representation Learning
• When Do Discourse Markers Affect Computational Sentence Understanding?
• Improving Sentence Representations with Consensus Maximisation
• Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
• Discourse Relation Embeddings: Representing the Relations between Discourse Segments in Social Media
• DiscSense: Automated Semantic Analysis of Discourse Markers
• Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations
• WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse
• PAUSE: Positive and Annealed Unlabeled Sentence Embedding
• Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference
• Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
• On Tree-Based Neural Sentence Modeling
• Semantically Driven Sentence Fusion: Modeling and Evaluation
• Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning
• Predictions For Pre-training Language Models
• Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification