Sentence Ordering and Coherence Modeling using Recurrent Neural Networks

Abstract

Modeling the structure of coherent texts is a key NLP problem. The task of coherently organizing a given set of sentences has been commonly used to build and evaluate models that understand such structure. We propose an end-to-end unsupervised deep learning approach based on the set-to-sequence framework to address this problem. Our model strongly outperforms prior methods in the order discrimination task and a novel task of ordering abstracts from scientific articles. Furthermore, our work shows that useful text representations can be obtained by learning to order sentences. Visualizing the learned sentence representations shows that the model captures high-level logical structure in paragraphs. Our representations perform comparably to state-of-the-art pre-training methods on sentence similarity and paraphrase detection tasks.

Cited by in corpus (16)

• Graph-based Neural Sentence Ordering
• Neural Sentence Ordering Based on Constraint Graphs
• Set-to-Sequence Methods in Machine Learning: a Review
• Local and Global Context-Based Pairwise Models for Sentence Ordering
• A Cross-Domain Transferable Neural Coherence Model
• Enhancing Semantic Understanding with Self-supervised Methods for Abstractive Dialogue Summarization
• Towards Modelling Coherence in Spoken Discourse
• InsertGNN: Can Graph Neural Networks Outperform Humans in TOEFL Sentence Insertion Problem?
• Learning Sentence Embeddings for Coherence Modelling and Beyond
• BERT4SO: Neural Sentence Ordering by Fine-tuning BERT
• Learning Better Representation for Tables by Self-Supervised Tasks
• Picking Apart Story Salads
• STaCK: Sentence Ordering with Temporal Commonsense Knowledge
• Improving Graph-based Sentence Ordering with Iteratively Predicted Pairwise Orderings
• Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers
• Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test