Transformer-based Conditional Variational Autoencoder for Controllable Story Generation

Abstract

We investigate large-scale latent variable models (LVMs) for neural story generation -- an under-explored application for open-domain long text -- with objectives in two threads: generation effectiveness and controllability. LVMs, especially the variational autoencoder (VAE), have achieved both effective and controllable generation through exploiting flexible distributional latent representations. Recently, Transformers and its variants have achieved remarkable effectiveness without explicit latent representation learning, thus lack satisfying controllability in generation. In this paper, we advocate to revive latent variable modeling, essentially the power of representation learning, in the era of Transformers to enhance controllability without hurting state-of-the-art generation effectiveness. Specifically, we integrate latent representation vectors with a Transformer-based pre-trained architecture to build conditional variational autoencoder (CVAE). Model components such as encoder, decoder and the variational posterior are all built on top of pre-trained language models -- GPT2 specifically in this paper. Experiments demonstrate state-of-the-art conditional generation ability of our model, as well as its excellent representation learning capability and controllability.

References in corpus (13)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Style Transfer from Non-Parallel Text by Cross-Alignment
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• Non-Autoregressive Machine Translation with Disentangled Context Transformer
• Variational Transformers for Diverse Response Generation
• Generative Neural Machine Translation
• Encoder-Agnostic Adaptation for Conditional Language Generation
• Non-Autoregressive Neural Dialogue Generation
• Multi-space Variational Encoder-Decoders for Semi-supervised Labeled Sequence Transduction
• Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events

Cited by in corpus (6)

• Privately Fine-Tuning Large Language Models with Differential Privacy
• PCAE: A Framework of Plug-in Conditional Auto-Encoder for Controllable Text Generation
• Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models
• Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generation
• Plug-and-Blend: A Framework for Controllable Story Generation with Blended Control Codes
• Cut the CARP: Fishing for zero-shot story evaluation