SciBERT: A Pretrained Language Model for Scientific Text

Abstract

Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github.com/allenai/scibert/.

arXiv commenthttps://github.com/allenai/scibert

References in corpus (6)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Publicly Available Clinical BERT Embeddings
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks
• Construction of the Literature Graph in Semantic Scholar
• From POS tagging to dependency parsing for biomedical event extraction

Cited by in corpus (30)

• Multi-Stage Document Ranking with BERT
• Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets
• Integrating Graph Contextualized Knowledge into Pre-trained Language Models
• Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
• A Survey of Knowledge Enhanced Pre-trained Models
• Membership Inference Attack Susceptibility of Clinical Language Models
• Rapid Adaptation of BERT for Information Extraction on Domain-Specific Business Documents
• BioFLAIR: Pretrained Pooled Contextualized Embeddings for Biomedical Sequence Labeling Tasks
• Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
• SINA-BERT: A pre-trained Language Model for Analysis of Medical Texts in Persian
• E-Stitchup: Data Augmentation for Pre-Trained Embeddings
• Pre-trained Language Model for Biomedical Question Answering
• Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
• Knowledge Guided Named Entity Recognition for BioMedical Text
• NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain
• Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
• Drug Repurposing for Cancer: An NLP Approach to Identify Low-Cost Therapies
• SciSummPip: An Unsupervised Scientific Paper Summarization Pipeline
• Parsing Early Modern English for Linguistic Search
• Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
• AGATHA: Automatic Graph-mining And Transformer based Hypothesis generation Approach
• Learning Better Sentence Representation with Syntax Information
• How Did This Get Funded?! Automatically Identifying Quirky Scientific Achievements
• Navigation-Based Candidate Expansion and Pretrained Language Models for Citation Recommendation
• Multimodal Graph-based Transformer Framework for Biomedical Relation Extraction
• Red Dragon AI at TextGraphs 2021 Shared Task: Multi-Hop Inference Explanation Regeneration by Matching Expert Ratings
• Text2Brain: Synthesis of Brain Activation Maps from Free-form Text Query
• An Evalutation of Programming Language Models' performance on Software Defect Detection
• SEAL: Scientific Keyphrase Extraction and Classification
• Assessing gender bias in medical and scientific masked language models with StereoSet