Pointing the Unknown Words

Abstract

The problem of rare and unknown words is an important issue that can potentially influence the performance of many NLP systems, including both the traditional count-based and the deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context.~We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known.~We observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset using our proposed model.

ACL 2016 Oral Paper

Cited by in corpus (40)

• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Selective Encoding for Abstractive Sentence Summarization
• A Survey of Knowledge-Enhanced Text Generation
• Neural Summarization by Extracting Sentences and Words
• Get To The Point: Summarization with Pointer-Generator Networks
• A Unified Query-based Generative Model for Question Generation and Question Answering
• Describing a Knowledge Base
• Controlling Output Length in Neural Encoder-Decoders
• Neural Question Generation from Text: A Preliminary Study
• Incorporating Discrete Translation Lexicons into Neural Machine Translation
• CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training
• A Graph-to-Sequence Model for AMR-to-Text Generation
• Data Recombination for Neural Semantic Parsing
• Query-Based Abstractive Summarization Using Neural Networks
• An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking
• Visual Question Answering with Memory-Augmented Networks
• Semantic Parsing with Syntax- and Table-Aware SQL Generation
• Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization
• Learning to Remember Translation History with a Continuous Cache
• A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
• Translating Phrases in Neural Machine Translation
• Reference-Aware Language Models
• Online Segment to Segment Neural Transduction
• Improving Neural Question Generation using Answer Separation
• Building Dynamic Knowledge Graphs from Text-based Games
• NAPS: Natural Program Synthesis Dataset
• Learning to Attend, Copy, and Generate for Session-Based Query Suggestion
• Sequential Copying Networks
• Closed-Book Training to Improve Summarization Encoder Memory
• Learning to Discriminate Noises for Incorporating External Information in Neural Machine Translation
• Neural Open Information Extraction
• Deep Conversational Recommender Systems: A New Frontier for Goal-Oriented Dialogue Systems
• Tackling Graphical NLP problems with Graph Recurrent Networks
• Category Aware Explainable Conversational Recommendation
• Reinforced Generative Adversarial Network for Abstractive Text Summarization
• Semantic Parsing with Dual Learning
• Multi-Source Pointer Network for Product Title Summarization
• Pointing to Subwords for Generating Function Names in Source Code
• News Article Teaser Tweets and How to Generate Them
• Pragmatic approach to structured data querying via natural language interface