Evaluating Document Coherence Modelling

Abstract

While pretrained language models ("LM") have driven impressive gains over morpho-syntactic and semantic tasks, their ability to model discourse and pragmatic phenomena is less clear. As a step towards a better understanding of their discourse modelling capabilities, we propose a sentence intrusion detection task. We examine the performance of a broad range of pretrained LMs on this detection task for English. Lacking a dataset for the task, we introduce INSteD, a novel intruder sentence detection dataset, containing 170,000+ documents constructed from English Wikipedia and CNN news articles. Our experiments show that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalisation capacity. Further results over a novel linguistic probe dataset show that there is substantial room for improvement, especially in the cross-domain setting.

arXiv commentaccepted to TACL 2021

References in corpus (15)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• A large annotated corpus for learning natural language inference
• Deep Text Classification Can be Fooled
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• A Simple Method for Commonsense Reasoning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• What do Neural Machine Translation Models Learn about Morphology?
• Towards Crafting Text Adversarial Samples
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Interpretable Adversarial Perturbation in Input Embedding Space for Text
• Designing and Interpreting Probes with Control Tasks
• Neural Discourse Structure for Text Categorization
• A Cross-Domain Transferable Neural Coherence Model
• CITE: A Corpus of Image-Text Discourse Relations