DisSent: Sentence Representation Learning from Explicit Discourse Relations
arXiv:1710.04334
Abstract
Learning effective representations of sentences is one of the core missions of natural language understanding. Existing models either train on a vast amount of text, or require costly, manually curated sentence relation datasets. We show that with dependency parsing and rule-based rubrics, we can curate a high quality sentence relation task by leveraging explicit discourse relations. We show that our curated dataset provides an excellent signal for learning vector representations of sentence meaning, representing relations that can only be determined when the meanings of two sentences are combined. We demonstrate that the automatically curated corpus allows a bidirectional LSTM sentence encoder to yield high quality sentence embeddings and can serve as a supervised fine-tuning dataset for larger models such as BERT. Our fixed sentence embeddings achieve high performance on a variety of transfer tasks, including SentEval, and we achieve state-of-the-art results on Penn Discourse Treebank's implicit relation prediction task.
13 pages, 4 figures. ACL 2019
References in corpus (4)
Cited by in corpus (22)
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
- Correlating neural and symbolic representations of language
- Deep Learning-based Sentiment Classification: A Comparative Survey
- Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
- Learning Compressed Sentence Representations for On-Device Text Processing
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
- Multi-view Sentence Representation Learning
- Neural Skill Transfer from Supervised Language Tasks to Reading Comprehension
- Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
- Learning to Explain: Answering Why-Questions via Rephrasing
- CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model
- Improving Sentence Representations with Consensus Maximisation
- Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
- Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference
- WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse
- Exploiting Invertible Decoders for Unsupervised Sentence Representation Learning
- Imbalanced Sentiment Classification Enhanced with Discourse Marker
- TransSent: Towards Generation of Structured Sentences with Discourse Marker
- NLITrans at SemEval-2018 Task 12: Transfer of Semantic Knowledge for Argument Comprehension
- Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units
- Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification