FinBERT: A Pretrained Language Model for Financial Communications
arXiv:2006.08097
Abstract
Contextual pretrained language models, such as BERT (Devlin et al., 2019), have made significant breakthrough in various NLP tasks by training on large scale of unlabeled text re-sources.Financial sector also accumulates large amount of financial communication text.However, there is no pretrained finance specific language models available. In this work,we address the need by pretraining a financial domain specific BERT models, FinBERT, using a large scale of financial communication corpora. Experiments on three financial sentiment classification tasks confirm the advantage of FinBERT over generic domain BERT model. The code and pretrained models are available at https://github.com/yya518/FinBERT. We hope this will be useful for practitioners and researchers working on financial NLP tasks.
https://github.com/yya518/FinBERT
References in corpus (4)
- Distributed Representations of Words and Phrases and their Compositionality
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
- Horovod: fast and easy distributed deep learning in TensorFlow
Cited by in corpus (10)
- FiNER: Financial Numeric Entity Recognition for XBRL Tagging
- AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
- Towards Automated Regulatory Compliance Verification in Financial Auditing with Large Language Models
- CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts
- Geo-FuB: A Method for Constructing an Operator-Function Knowledge Base for Geospatial Code Generation Tasks Using Large Language Models
- Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models
- JuriBERT: A Masked-Language Model Adaptation for French Legal Text
- Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
- Zero-shot Task Transfer for Invoice Extraction via Class-aware QA Ensemble
- DICoE@FinSim-3: Financial Hypernym Detection using Augmented Terms and Distance-based Features