Which Discriminator for Cooperative Text Generation?

Abstract

Language models generate texts by successively predicting probability distributions for next tokens given past ones. A growing field of interest tries to leverage external information in the decoding process so that the generated texts have desired properties, such as being more natural, non toxic, faithful, or having a specific writing style. A solution is to use a classifier at each generation step, resulting in a cooperative environment where the classifier guides the decoding of the language model distribution towards relevant texts for the task at hand. In this paper, we examine three families of (transformer-based) discriminators for this specific task of cooperative decoding: bidirectional, left-to-right and generative ones. We evaluate the pros and cons of these different types of discriminators for cooperative generation, exploring respective accuracy on classification tasks along with their impact on the resulting sample quality and computational performances. We also provide the code of a batched implementation of the powerful cooperative decoding strategy used for our experiments, the Monte Carlo Tree Search, working with each discriminator for Natural Language Generation.

arXiv comment6 pages, 2 figures, accepted to SIGIR 2022

References in corpus (9)

• Decoupled Weight Decay Regularization
• Character-level Convolutional Networks for Text Classification
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• The Curious Case of Neural Text Degeneration
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Residual Energy-Based Models for Text
• Discriminative Adversarial Search for Abstractive Summarization
• Adding A Filter Based on The Discriminator to Improve Unconditional Text Generation