Learning Sentence Embeddings for Coherence Modelling and Beyond
arXiv:1804.08053
Abstract
We present a novel and effective technique for performing text coherence tasks while facilitating deeper insights into the data. Despite obtaining ever-increasing task performance, modern deep-learning approaches to NLP tasks often only provide users with the final network decision and no additional understanding of the data. In this work, we show that a new type of sentence embedding learned through self-supervision can be applied effectively to text coherence tasks while serving as a window through which deeper understanding of the data can be obtained. To produce these sentence embeddings, we train a recurrent neural network to take individual sentences and predict their location in a document in the form of a distribution over locations. We demonstrate that these embeddings, combined with simple visual heuristics, can be used to achieve performance competitive with state-of-the-art on multiple text coherence tasks, outperforming more complex and specialized approaches. Additionally, we demonstrate that these embeddings can provide insights useful to writers for improving writing quality and informing document structuring, and assisting readers in summarizing and locating information.
arXiv commentAccepted for publication at RANLP 2019. 8 pages (10 with references), 4 Figures in the main text. This version contains significant improvements in the algorithm and reports on a wider set of applications
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Efficient Estimation of Word Representations in Vector Space
- Teaching Machines to Read and Comprehend
- Empirical Evaluation of Rectified Activations in Convolutional Network
- A Deep Reinforced Model for Abstractive Summarization
- SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- Inferring Strategies for Sentence Ordering in Multidocument News Summarization
- Representation learning for very short texts using weighted word embedding aggregation
- Order Matters: Sequence to sequence for sets
- Neural Summarization by Extracting Sentences and Words
- Classify or Select: Neural Architectures for Extractive Document Summarization
- Distraction-Based Neural Networks for Document Summarization
- Sentence Ordering and Coherence Modeling using Recurrent Neural Networks
- End-to-End Neural Sentence Ordering Using Pointer Network
- Neural Sentence Ordering
- Text Coherence Analysis Based on Deep Neural Network