A Hierarchical Neural Autoencoder for Paragraphs and Documents
arXiv:1506.01057
Abstract
Natural language generation of coherent long texts like paragraphs or longer documents is a challenging problem for recurrent networks models. In this paper, we explore an important step toward this generation task: training an LSTM (Long-short term memory) auto-encoder to preserve and reconstruct multi-sentence paragraphs. We introduce an LSTM model that hierarchically builds an embedding for a paragraph from embeddings for sentences and words, then decodes this embedding to reconstruct the original paragraph. We evaluate the reconstructed paragraph using standard metrics like ROUGE and Entity Grid, showing that neural models are able to encode texts in a way that preserve syntactic, semantic, and discourse coherence. While only a first step toward generating coherent text units from neural models, our work has the potential to significantly impact natural language generation and summarization\footnote{Code for the three models described in this paper can be found at www.stanford.edu/~jiweil/ .
References in corpus (4)
Cited by in corpus (85)
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection
- A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
- Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
- A Simple, Fast Diverse Decoding Algorithm for Neural Generation
- Adversarial Learning for Neural Dialogue Generation
- Tweet2Vec: Learning Tweet Embeddings Using Character-level CNN-LSTM Encoder-Decoder
- Towards Universal Paraphrastic Sentence Embeddings
- Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
- Mutual Information and Diverse Decoding Improve Neural Machine Translation
- Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems
- Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
- Fonduer: Knowledge Base Construction from Richly Formatted Data
- StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow
- Deconvolutional Paragraph Representation Learning
- Document Context Language Models
- Distraction-Based Neural Networks for Document Summarization
- DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
- Generating Multi-Agent Trajectories using Programmatic Weak Supervision
- Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation
- Stacked Cross Attention for Image-Text Matching
- Dual Attention Networks for Multimodal Reasoning and Matching
- Video-based Sign Language Recognition without Temporal Segmentation
- Feature Selection and Feature Extraction in Pattern Analysis: A Literature Review
- Similarity Learning for Authorship Verification in Social Media
- IcoRating: A Deep-Learning System for Scam ICO Identification
- Neural Emoji Recommendation in Dialogue Systems
- Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
- Deep Learning in Bioinformatics
- Word Mover's Embedding: From Word2Vec to Document Embedding
- Zero-Shot Dialog Generation with Cross-Domain Latent Actions
- Learning Generic Sentence Representations Using Convolutional Neural Networks
- Neural Net Models for Open-Domain Discourse Coherence
- The Winning Solution to the IEEE CIG 2017 Game Data Mining Competition
- User Modeling for Task Oriented Dialogues
- A Hierarchical Approach for Generating Descriptive Image Paragraphs
- Generating Distractors for Reading Comprehension Questions from Real Examinations
- One-Class Adversarial Nets for Fraud Detection
- A Hierarchical Recurrent Neural Network for Symbolic Melody Generation
- An Empirical Evaluation of Visual Question Answering for Novel Objects
- Natural Language Generation with Neural Variational Models
- Tree Memory Networks for Modelling Long-term Temporal Dependencies
- A Hierarchical Network for Abstractive Meeting Summarization with Cross-Domain Pretraining
- Learning Distributed Representations of Texts and Entities from Knowledge Base
- Toward Extractive Summarization of Online Forum Discussions via Hierarchical Attention Networks
- Learning to Attend, Copy, and Generate for Session-Based Query Suggestion
- Recurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning
- Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation
- A Survey of Neural Network Techniques for Feature Extraction from Text
- What Makes A Good Story? Designing Composite Rewards for Visual Storytelling
- Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents
- Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
- Improving Textual Network Embedding with Global Attention via Optimal Transport
- A Planning based Framework for Essay Generation
- DocTag2Vec: An Embedding Based Multi-label Learning Approach for Document Tagging
- Towards Abstraction from Extraction: Multiple Timescale Gated Recurrent Unit for Summarization
- Denoising based Sequence-to-Sequence Pre-training for Text Generation
- Outline Generation: Understanding the Inherent Content Structure of Documents
- Path-Based Contextualization of Knowledge Graphs for Textual Entailment
- Data-Space Inversion Using a Recurrent Autoencoder for Time-Series Parameterization
- Learning Bilingual Sentence Embeddings via Autoencoding and Computing Similarities with a Multilayer Perceptron
- The Style-Content Duality of Attractiveness: Learning to Write Eye-Catching Headlines via Disentanglement
- Assessing the Linguistic Productivity of Unsupervised Deep Neural Networks
- Query2Vec: An Evaluation of NLP Techniques for Generalized Workload Analytics
- Why and How to Pay Different Attention to Phrase Alignments of Different Intensities
- Observing Dialogue in Therapy: Categorizing and Forecasting Behavioral Codes
- Style Obfuscation by Invariance
- Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms
- Autoencoders as Tools for Program Synthesis
- Toward the Understanding of Deep Text Matching Models for Information Retrieval
- A Variational Approach to Unsupervised Sentiment Analysis
- Learning to Embed Words in Context for Syntactic Tasks
- Co-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehension
- On the notion of number in humans and machines
- EgoCoder: Intelligent Program Synthesis with Hierarchical Sequential Neural Network Model
- Towards Controllable and Personalized Review Generation
- Building Context-aware Clause Representations for Situation Entity Type Classification
- Dynamic Graph Embedding via LSTM History Tracking
- OhioState at IJCNLP-2017 Task 4: Exploring Neural Architectures for Multilingual Customer Feedback Analysis
- Linguistic Versus Latent Relations for Modeling Coherent Flow in Paragraphs
- Hierarchical RNN with Static Sentence-Level Attention for Text-Based Speaker Change Detection
- Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification
- Ranking sentences from product description & bullets for better search
- Recommender Systems with Heterogeneous Side Information
- An Unsupervised Model with Attention Autoencoders for Question Retrieval