Encoder-Agnostic Adaptation for Conditional Language Generation

Abstract

Large pretrained language models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that adapt a pretrained model for arbitrary downstream tasks. However it is an open-question how to use similar techniques for language generation. Early results in the encoder-agnostic setting have been mostly negative. In this work we explore methods for adapting a pretrained language model to arbitrary conditional input. We observe that pretrained transformer models are sensitive to large parameter changes during tuning. We therefore propose an adaptation that directly injects arbitrary conditioning into self attention, an approach we call pseudo self attention. Through experiments on four diverse conditional text generation tasks we show that this encoder-agnostic technique outperforms strong baselines, produces coherent generations, and is data efficient.

References in corpus (14)

• Distributed Representations of Words and Phrases and their Compositionality
• Teaching Machines to Read and Comprehend
• Cross-lingual Language Model Pretraining
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Style Transfer from Non-Parallel Text by Cross-Alignment
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• On Using Monolingual Corpora in Neural Machine Translation
• Measuring the Effects of Data Parallelism on Neural Network Training
• Hierarchical Neural Story Generation
• Parameter-Efficient Transfer Learning for NLP
• Language Models with Transformers
• Pre-trained Language Model Representations for Language Generation
• The Neural Noisy Channel
• An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

Cited by in corpus (10)

• Transformer-based Conditional Variational Autoencoder for Controllable Story Generation
• Multimodal Conditionality for Natural Language Generation
• Emotional Neural Language Generation Grounded in Situational Contexts
• Conditional Text Generation for Harmonious Human-Machine Interaction
• Align then Summarize: Automatic Alignment Methods for Summarization Corpus Creation
• Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events
• Improving Language Generation with Sentence Coherence Objective
• Unsupervised Pre-training for Natural Language Generation: A Literature Review
• Improvement of a dedicated model for open domain persona-aware dialogue generation
• Attribute Alignment: Controlling Text Generation from Pre-trained Language Models