Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
arXiv:2010.05906
Abstract
Abductive and counterfactual reasoning, core abilities of everyday human cognition, require reasoning about what might have happened at time t, while conditioning on multiple contexts from the relative past and future. However, simultaneous incorporation of past and future contexts using generative language models (LMs) can be challenging, as they are trained either to condition only on the past context or to perform narrowly scoped text-infilling. In this paper, we propose DeLorean, a new unsupervised decoding algorithm that can flexibly incorporate both the past and future contexts using only off-the-shelf, left-to-right language models and no supervision. The key intuition of our algorithm is incorporating the future through back-propagation, during which, we only update the internal representation of the output while fixing the model parameters. By alternating between forward and backward propagation, DeLorean can decode the output representation that reflects both the left and right contexts. We demonstrate that our approach is general and applicable to two nonmonotonic reasoning tasks: abductive text generation and counterfactual story revision, where DeLorean outperforms a range of unsupervised and some supervised methods, based on automatic and human evaluation.
arXiv commentEMNLP 2020
References in corpus (11)
- BERTScore: Evaluating Text Generation with BERT
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Style Transfer from Non-Parallel Text by Cross-Alignment
- A large annotated corpus for learning natural language inference
- Phrase-Based & Neural Unsupervised Machine Translation
- Plug and Play Language Models: A Simple Approach to Controlled Text Generation
- A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories
- Unsupervised Neural Machine Translation
- Abductive Commonsense Reasoning
- Generating Sentences from Disentangled Syntactic and Semantic Spaces
- WIQA: A dataset for "What if..." reasoning over procedural text