A Comparative Study of Pretrained Language Models for Long Clinical Text

Abstract

Objective: Clinical knowledge enriched transformer models (e.g., ClinicalBERT) have state-of-the-art results on clinical NLP (natural language processing) tasks. One of the core limitations of these transformer models is the substantial memory consumption due to their full self-attention mechanism, which leads to the performance degradation in long clinical texts. To overcome this, we propose to leverage long-sequence transformer models (e.g., Longformer and BigBird), which extend the maximum input sequence length from 512 to 4096, to enhance the ability to model long-term dependencies in long clinical texts. Materials and Methods: Inspired by the success of long sequence transformer models and the fact that clinical notes are mostly long, we introduce two domain enriched language models, Clinical-Longformer and Clinical-BigBird, which are pre-trained on a large-scale clinical corpus. We evaluate both language models using 10 baseline tasks including named entity recognition, question answering, natural language inference, and document classification tasks. Results: The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT and other short-sequence transformers in all 10 downstream tasks and achieve new state-of-the-art results. Discussion: Our pre-trained language models provide the bedrock for clinical NLP using long texts. We have made our source code available at https://github.com/luoyuanlab/Clinical-Longformer, and the pre-trained models available for public download at: https://huggingface.co/yikuan8/Clinical-Longformer. Conclusion: This study demonstrates that clinical knowledge enriched long-sequence transformers are able to learn long-term dependencies in long clinical text. Our methods can also inspire the development of other domain-enriched long-sequence transformers.

arXiv commentarXiv admin note: substantial text overlap with arXiv:2201.11838

References in corpus (8)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Longformer: The Long-Document Transformer
• Publicly Available Clinical BERT Embeddings
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• KG-BERT: BERT for Knowledge Graph Completion
• CollaboNet: collaboration of deep neural networks for biomedical named entity recognition
• Adapting and evaluating a deep learning language model for clinical why-question answering
• CancerBERT: a BERT model for Extracting Breast Cancer Phenotypes from Electronic Health Records

Cited by in corpus (5)

• Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Models
• EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation
• Large Language Models for Healthcare Text Classification: A Systematic Review
• Clinical information extraction for Low-resource languages with Few-shot learning using Pre-trained language models and Prompting
• Enhanced Labeling Technique for Reddit Text and Fine-Tuned Longformer Models for Classifying Depression Severity in English and Luganda