Extractive Summary as Discrete Latent Variables

Abstract

In this paper, we compare various methods to compress a text using a neural model. We find that extracting tokens as latent variables significantly outperforms the state-of-the-art discrete latent variable models such as VQ-VAE. Furthermore, we compare various extractive compression schemes. There are two best-performing methods that perform equally. One method is to simply choose the tokens with the highest tf-idf scores. Another is to train a bidirectional language model similar to ELMo and choose the tokens with the highest loss. If we consider any subsequence of a text to be a text in a broader sense, we conclude that language is a strong compression code of itself. Our finding justifies the high quality of generation achieved with hierarchical method, as their latent variables are nothing but natural language summary. We also conclude that there is a hierarchy in language such that an entire text can be predicted much more easily based on a sequence of a small number of keywords, which can be easily found by classical methods as tf-idf. We speculate that this extraction process may be useful for unsupervised hierarchical text generation.

References in corpus (25)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Neural Discrete Representation Learning
• Improved Training of Wasserstein GANs
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Efficient Neural Architecture Search via Parameter Sharing
• Achieving Human Parity on Automatic Chinese to English News Translation
• Deep Learning Scaling is Predictable, Empirically
• MaskGAN: Better Text Generation via Filling in the______
• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Long Text Generation via Adversarial Training with Leaked Information
• Texygen: A Benchmarking Platform for Text Generation Models
• Hierarchical Neural Story Generation
• Toward Controlled Generation of Text
• Adversarial Feature Matching for Text Generation
• The challenge of realistic music generation: modelling raw audio at scale
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training
• Language GANs Falling Short
• On Accurate Evaluation of GANs for Language Generation
• Theory and Experiments on Vector Quantized Autoencoders
• Analyzing Uncertainty in Neural Machine Translation
• Adaptive Input Representations for Neural Language Modeling
• Language as a Latent Variable: Discrete Generative Models for Sentence Compression
• Discrete Autoencoders for Sequence Models
• Hierarchical Text Generation using an Outline