HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

Abstract

Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods. Training the hierarchical encoder with these \emph{inaccurate} labels is challenging. Inspired by the recent work on pre-training transformer sentence encoders \cite{devlin:2018:arxiv}, we propose {\sc Hibert} (as shorthand for {\bf HI}erachical {\bf B}idirectional {\bf E}ncoder {\bf R}epresentations from {\bf T}ransformers) for document encoding and a method to pre-train it using unlabeled data. We apply the pre-trained {\sc Hibert} to our summarization model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times dataset. We also achieve the state-of-the-art performance on these two datasets.

to appear in ACL 2019

References in corpus (2)

• Sequence to Sequence Learning with Neural Networks
• SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents

Cited by in corpus (10)

• Toward Transformer-Based Object Detection
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• Recursively Summarizing Books with Human Feedback
• Code-switched inspired losses for generic spoken dialog representations
• A Sentence-level Hierarchical BERT Model for Document Classification with Limited Labelled Data
• Fact-level Extractive Summarization with Hierarchical Graph Mask on BERT
• Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations
• Hierarchical GPT with Congruent Transformers for Multi-Sentence Language Models
• AGenT Zero: Zero-shot Automatic Multiple-Choice Question Generation for Skill Assessments
• Improving Abstractive Text Summarization with History Aggregation