HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

Abstract

Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods. Training the hierarchical encoder with these \emph{inaccurate} labels is challenging. Inspired by the recent work on pre-training transformer sentence encoders \cite{devlin:2018:arxiv}, we propose {\sc Hibert} (as shorthand for {\bf HI}erachical {\bf B}idirectional {\bf E}ncoder {\bf R}epresentations from {\bf T}ransformers) for document encoding and a method to pre-train it using unlabeled data. We apply the pre-trained {\sc Hibert} to our summarization model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times dataset. We also achieve the state-of-the-art performance on these two datasets.

arXiv commentto appear in ACL 2019

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Teaching Machines to Read and Comprehend
• A Deep Reinforced Model for Abstractive Summarization
• SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Neural Extractive Text Summarization with Syntactic Compression

Cited by in corpus (18)

• Natural Language Processing Advancements By Deep Learning: A Survey
• Toward Transformer-Based Object Detection
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching
• Recursively Summarizing Books with Human Feedback
• ETC: Encoding Long and Structured Inputs in Transformers
• Learning to Encode Position for Transformer with Continuous Dynamical Model
• Code-switched inspired losses for generic spoken dialog representations
• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
• A Sentence-level Hierarchical BERT Model for Document Classification with Limited Labelled Data
• SciSummPip: An Unsupervised Scientific Paper Summarization Pipeline
• Fact-level Extractive Summarization with Hierarchical Graph Mask on BERT
• Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations
• AGenT Zero: Zero-shot Automatic Multiple-Choice Question Generation for Skill Assessments
• Iterative Decoding for Compositional Generalization in Transformers
• Hierarchical GPT with Congruent Transformers for Multi-Sentence Language Models
• Improving Abstractive Text Summarization with History Aggregation
• Corruption Is Not All Bad: Incorporating Discourse Structure into Pre-training via Corruption for Essay Scoring