Examining the rhetorical capacities of neural language models

Abstract

Recently, neural language models (LMs) have demonstrated impressive abilities in generating high-quality discourse. While many recent papers have analyzed the syntactic aspects encoded in LMs, there has been no analysis to date of the inter-sentential, rhetorical knowledge. In this paper, we propose a method that quantitatively evaluates the rhetorical capacities of neural LMs. We examine the capacities of neural LMs understanding the rhetoric of discourse by evaluating their abilities to encode a set of linguistic features derived from Rhetorical Structure Theory (RST). Our experiments show that BERT-based LMs outperform other Transformer LMs, revealing the richer discourse knowledge in their intermediate layer representations. In addition, GPT-2 and XLNet apparently encode less rhetorical knowledge, and we suggest an explanation drawing from linguistic philosophy. Our method shows an avenue towards quantifying the rhetorical capacities of neural LMs.

arXiv commentEMNLP 2020 BlackboxNLP Workshop

References in corpus (7)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Cross-lingual Language Model Pretraining
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Understanding intermediate layers using linear classifier probes
• Fast Rhetorical Structure Theory Discourse Parsing