Pretrained Language Models for Sequential Sentence Classification

Abstract

As a step toward better document-level understanding, we explore classification of a sequence of sentences into their corresponding categories, a task that requires understanding sentences in context of the document. Recent successful models for this task have used hierarchical models to contextualize sentence representations, and Conditional Random Fields (CRFs) to incorporate dependencies between subsequent labels. In this work, we show that pretrained language models, BERT (Devlin et al., 2018) in particular, can be used for this task to capture contextual dependencies without the need for hierarchical encoding nor a CRF. Specifically, we construct a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences. Our approach achieves state-of-the-art results on four datasets, including a new dataset of structured scientific abstracts.

EMNLP 2019

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Teaching Machines to Read and Comprehend
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
• Neural Summarization by Extracting Sentences and Words
• Language Model Pre-training for Hierarchical Document Representations

Cited by in corpus (15)

• Bringing order into the realm of Transformer-based language models for artificial intelligence and law
• Improving Document-Level Sentiment Classification Using Importance of Sentences
• Comprehensive evaluation of Mal-API-2019 dataset by machine learning in malware detection
• PersonaFlow: Designing LLM-Simulated Expert Perspectives for Enhanced Research Ideation
• Generating a Structured Summary of Numerous Academic Papers: Dataset and Method
• Requirements Analysis for an Open Research Knowledge Graph
• MS2: Multi-Document Summarization of Medical Studies
• On the Trade-off between Redundancy and Local Coherence in Summarization
• On Generating Extended Summaries of Long Documents
• Classification and Clustering of Sentence-Level Embeddings of Scientific Articles Generated by Contrastive Learning
• Look Ahead Text Understanding and LLM Stitching
• Multi-turn Dialogue Reading Comprehension with Pivot Turns and Knowledge
• Efficient Domain Adaptation of Language Models via Adaptive Tokenization
• Explainable Prediction of Text Complexity: The Missing Preliminaries for Text Simplification
• Unsupervised Extractive Summarization by Human Memory Simulation