Federated pretraining and fine tuning of BERT using clinical notes from multiple silos
arXiv:2002.08562
Abstract
Large scale contextual representation models, such as BERT, have significantly advanced natural language processing (NLP) in recently years. However, in certain area like healthcare, accessing diverse large scale text data from multiple institutions is extremely challenging due to privacy and regulatory reasons. In this article, we show that it is possible to both pretrain and fine tune BERT models in a federated manner using clinical texts from different silos without moving the data.
References in corpus (5)
- Distributed Representations of Words and Phrases and their Compositionality
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Federated Learning: Strategies for Improving Communication Efficiency
- Enhancing Clinical Concept Extraction with Contextual Embeddings