Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network

Abstract

Capturing the compositional process which maps the meaning of words to that of documents is a central challenge for researchers in Natural Language Processing and Information Retrieval. We introduce a model that is able to represent the meaning of documents by embedding them in a low dimensional vector space, while preserving distinctions of word and sentence order crucial for capturing nuanced semantics. Our model is based on an extended Dynamic Convolution Neural Network, which learns convolution filters at both the sentence and document level, hierarchically learning to capture and compose low level lexical features into high level semantic concepts. We demonstrate the effectiveness of this model on a range of document modelling tasks, achieving strong results with no feature engineering and with a more compact model. Inspired by recent advances in visualising deep convolution networks for computer vision, we present a novel visualisation technique for our document networks which not only provides insight into their learning process, but also can be interpreted to produce a compelling automatic summarisation system for texts.

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• Natural Language Processing (almost) from Scratch
• Distributed Representations of Sentences and Documents
• Visualizing and Understanding Convolutional Networks
• Experimental Support for a Categorical Compositional Distributional Model of Meaning
• Knowledge Matters: Importance of Prior Information for Optimization
• Training Restricted Boltzmann Machines on Word Observations

Cited by in corpus (25)

• Recent Trends in Deep Learning Based Natural Language Processing
• A C-LSTM Neural Network for Text Classification
• A Convolutional Attention Network for Extreme Summarization of Source Code
• Deep Learning applied to NLP
• From Captions to Visual Concepts and Back
• Neural Models for Information Retrieval
• Document Context Language Models
• Modeling Relational Information in Question-Answer Pairs with Convolutional Neural Networks
• Reader-Aware Multi-Document Summarization via Sparse Coding
• A Comparison of Machine Learning Algorithms for the Surveillance of Autism Spectrum Disorder
• Abstractive Multi-Document Summarization via Phrase Selection and Merging
• Deep Multi-Task Learning with Shared Memory
• Multichannel Variable-Size Convolution for Sentence Classification
• Compositionality decomposed: how do neural networks generalise?
• Deep Multi-Instance Transfer Learning
• Feature Weight Tuning for Recursive Neural Networks
• Dynamic Compositional Neural Networks over Tree Structure
• Cached Long Short-Term Memory Neural Networks for Document-Level Sentiment Classification
• Open-domain Event Extraction and Embedding for Natural Gas Market Prediction
• Toward the Understanding of Deep Text Matching Models for Information Retrieval
• Recurrent Point Review Models
• Weakly Supervised Domain Detection
• Joint Learning of Distributed Representations for Images and Texts
• Task-specific Word Identification from Short Texts Using a Convolutional Neural Network
• Understanding Convolutional Neural Networks with A Mathematical Model