Incorporating Copying Mechanism in Sequence-to-Sequence Learning

Abstract

We address an important problem in sequence-to-sequence (Seq2Seq) learning referred to as copying, in which certain segments in the input sequence are selectively replicated in the output sequence. A similar phenomenon is observable in human language communication. For example, humans tend to repeat entity names or even long phrases in conversation. The challenge with regard to copying in Seq2Seq is that new machinery is needed to decide when to perform the operation. In this paper, we incorporate copying into neural network-based Seq2Seq learning and propose a new model called CopyNet with encoder-decoder structure. CopyNet can nicely integrate the regular way of word generation in the decoder with the new copying mechanism which can choose sub-sequences in the input sequence and put them at proper places in the output sequence. Our empirical study on both synthetic data sets and real world data sets demonstrates the efficacy of CopyNet. For example, CopyNet can outperform regular RNN-based model with remarkable margins on text summarization tasks.

10 pages, 5 figures, accepted by ACL2016

References in corpus (6)

• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A Neural Conversational Model
• Neural Summarization by Extracting Sentences and Words
• A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
• Minimum Risk Training for Neural Machine Translation

Cited by in corpus (97)

• Machine Comprehension Using Match-LSTM and Answer Pointer
• Selective Encoding for Abstractive Sentence Summarization
• Neural Summarization by Extracting Sentences and Words
• Transforming Question Answering Datasets Into Natural Language Inference Datasets
• Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
• A Neural Knowledge Language Model
• A Syntactic Neural Model for General-Purpose Code Generation
• Get To The Point: Summarization with Pointer-Generator Networks
• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
• Neural Machine Translation with External Phrase Memory
• A Unified Query-based Generative Model for Question Generation and Question Answering
• Semi-Supervised Variational Reasoning for Medical Dialogue Generation
• Actor-Critic based Training Framework for Abstractive Summarization
• Grammar-based Neural Text-to-SQL Generation
• Describing a Knowledge Base
• Controlling Output Length in Neural Encoder-Decoders
• Neural Headline Generation with Sentence-wise Optimization
• Graph Transformer for Graph-to-Sequence Learning
• Table2Charts: Recommending Charts by Learning Shared Table Representations
• Deep Recurrent Generative Decoder for Abstractive Text Summarization
• A Graph-to-Sequence Model for AMR-to-Text Generation
• On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
• A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss
• Chat Detection in an Intelligent Assistant: Combining Task-oriented and Non-task-oriented Spoken Dialogue Systems
• Paper Abstract Writing through Editing Mechanism
• Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow
• Global Encoding for Abstractive Summarization
• Domain Adaptive Dialog Generation via Meta Learning
• An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking
• Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information
• A Semantic Relevance Based Neural Network for Text Summarization and Text Simplification
• A Transfer-Learnable Natural Language Interface for Databases
• Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusion
• Hierarchical LSTMs with Adaptive Attention for Visual Captioning
• RepeatNet: A Repeat Aware Neural Recommendation Machine for Session-based Recommendation
• PONE: A Novel Automatic Evaluation Metric for Open-Domain Generative Dialogue Systems
• A Hierarchical Framework for Relation Extraction with Reinforcement Learning
• Production Ready Chatbots: Generate if not Retrieve
• Semantic Parsing with Syntax- and Table-Aware SQL Generation
• Story Realization: Expanding Plot Events into Sentences
• Keyphrase Extraction with Span-based Feature Representations
• Steering Output Style and Topic in Neural Response Generation
• A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
• Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization
• Learning to Remember Translation History with a Continuous Cache
• Translating Phrases in Neural Machine Translation
• Neural Machine Translation Advised by Statistical Machine Translation
• Hierarchical Context Tagging for Utterance Rewriting
• Toward Code Generation: A Survey and Lessons from Semantic Parsing
• Operations Guided Neural Networks for High Fidelity Data-To-Text Generation
• Autoencoder as Assistant Supervisor: Improving Text Representation for Chinese Social Media Text Summarization
• Keyphrase Generation with Correlation Constraints
• Meaningful Answer Generation of E-Commerce Question-Answering
• Learning to Attend, Copy, and Generate for Session-Based Query Suggestion
• Retrieval-Based Neural Code Generation
• Thinking Globally, Acting Locally: Distantly Supervised Global-to-Local Knowledge Selection for Background Based Conversation
• RefNet: A Reference-aware Network for Background Based Conversation
• Story Ending Generation with Incremental Encoding and Commonsense Knowledge
• Enabling Automatic Repair of Source Code Vulnerabilities Using Data-Driven Methods
• Language Modeling with Sparse Product of Sememe Experts
• Sequential Copying Networks
• Improving Knowledge-aware Dialogue Generation via Knowledge Base Question Answering
• Enhancing Function Name Prediction using Votes-Based Name Tokenization and Multi-Task Learning
• Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency
• Improving Semantic Relevance for Sequence-to-Sequence Learning of Chinese Social Media Text Summarization
• Learning to Discriminate Noises for Incorporating External Information in Neural Machine Translation
• Interpretation of Natural Language Rules in Conversational Machine Reading
• Closed-Book Training to Improve Summarization Encoder Memory
• Upcycle Your OCR: Reusing OCRs for Post-OCR Text Correction in Romanised Sanskrit
• Be Concise and Precise: Synthesizing Open-Domain Entity Descriptions from Facts
• Encode, Tag, Realize: High-Precision Text Editing
• Unsupervised Learning of KB Queries in Task-Oriented Dialogs
• Title-Guided Encoding for Keyphrase Generation
• Data Augmentation for Copy-Mechanism in Dialogue State Tracking
• Knowledge Based Machine Reading Comprehension
• Question Generation from SQL Queries Improves Neural Semantic Parsing
• Semantic Graphs for Generating Deep Questions
• Product-Aware Answer Generation in E-Commerce Question-Answering
• Abstractive Text Summarization by Incorporating Reader Comments
• Multi-Source Pointer Network for Product Title Summarization
• Partially-Aligned Data-to-Text Generation with Distant Supervision
• Improving Social Media Text Summarization by Learning Sentence Weight Distribution
• PoMo: Generating Entity-Specific Post-Modifiers in Context
• Fact-aware Sentence Split and Rephrase with Permutation Invariant Training
• Translating Natural Language to SQL using Pointer-Generator Networks and How Decoding Order Matters
• A Sketch-Based Neural Model for Generating Commit Messages from Diffs
• GRET: Global Representation Enhanced Transformer
• Improving Abstractive Text Summarization with History Aggregation
• FANDA: A Novel Approach to Perform Follow-up Query Analysis
• Generating Highly Relevant Questions
• CN-Probase: A Data-driven Approach for Large-scale Chinese Taxonomy Construction
• End to End Dialogue Transformer
• Learning to Summarize Long Texts with Memory Compression and Transfer
• Learning to Generate Structured Queries from Natural Language with Indirect Supervision
• Keeping Notes: Conditional Natural Language Generation with a Scratchpad Mechanism
• Incorporating Relevant Knowledge in Context Modeling and Response Generation
• Tree Decomposition Attention for AMR-to-Text Generation