Sparse Text Generation

Abstract

Current state-of-the-art text generators build on powerful language models such as GPT-2, achieving impressive performance. However, to avoid degenerate text, they require sampling from a modified softmax, via temperature parameters or ad-hoc truncation techniques, as in top-$k$ or nucleus sampling. This creates a mismatch between training and testing conditions. In this paper, we use the recently introduced entmax transformation to train and sample from a natively sparse language model, avoiding this mismatch. The result is a text generator with favorable performance in terms of fluency and consistency, fewer repetitions, and n-gram diversity closer to human text. In order to evaluate our model, we propose three new metrics for comparing sparse or truncated distributions: $\epsilon$-perplexity, sparsemax score, and Jensen-Shannon divergence. Human-evaluated experiments in story completion and dialogue generation show that entmax sampling leads to more engaging and coherent stories and conversations.

References in corpus (9)

• Sequence to Sequence Learning with Neural Networks
• Attention-Based Models for Speech Recognition
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training
• Language GANs Falling Short
• ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons
• Eval all, trust a few, do wrong to none: Comparing sentence generation models