Point-less: More Abstractive Summarization with Pointer-Generator Networks

Abstract

The Pointer-Generator architecture has shown to be a big improvement for abstractive summarization seq2seq models. However, the summaries produced by this model are largely extractive as over 30% of the generated sentences are copied from the source text. This work proposes a multihead attention mechanism, pointer dropout, and two new loss functions to promote more abstractive summaries while maintaining similar ROUGE scores. Both the multihead attention and dropout do not improve N-gram novelty, however, the dropout acts as a regularizer which improves the ROUGE score. The new loss function achieves significantly higher novel N-grams and sentences, at the cost of a slightly lower ROUGE score.

7 pages

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Categorical Reparameterization with Gumbel-Softmax
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Get To The Point: Summarization with Pointer-Generator Networks
• Structure-Infused Copy Mechanisms for Abstractive Summarization
• Query Focused Abstractive Summarization: Incorporating Query Relevance, Multi-Document Coverage, and Summary Length Constraints into seq2seq Models
• Text Summarization using Abstract Meaning Representation
• Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information
• Controlling Decoding for More Abstractive Summaries with Copy-Based Networks
• Abstractive Summarization Using Attentive Neural Techniques