Portuguese Named Entity Recognition using BERT-CRF

Abstract

Recent advances in language representation using neural networks have made it viable to transfer the learned internal states of a trained model to downstream natural language processing tasks, such as named entity recognition (NER) and question answering. It has been shown that the leverage of pre-trained language models improves the overall performance on many tasks and is highly beneficial when labeled data is scarce. In this work, we train Portuguese BERT models and employ a BERT-CRF architecture to the NER task on the Portuguese language, combining the transfer capabilities of BERT with the structured predictions of CRF. We explore feature-based and fine-tuning training strategies for the BERT model. Our fine-tuning approach obtains new state-of-the-art results on the HAREM I dataset, improving the F1-score by 1 point on the selective scenario (5 NE classes) and by 4 points on the total scenario (10 NE classes).

References in corpus (4)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• A Survey on Recent Advances in Named Entity Recognition from Deep Learning models

Cited by in corpus (24)

• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• FlauBERT: Unsupervised Language Model Pre-training for French
• SemEval-2021 Task 11: NLPContributionGraph -- Structuring Scholarly NLP Contributions for a Research Knowledge Graph
• German BERT Model for Legal Named Entity Recognition
• Transformers: "The End of History" for NLP?
• FLERT: Document-Level Features for Named Entity Recognition
• PharmKE: Knowledge Extraction Platform for Pharmaceutical Texts using Transfer Learning
• Aspect and Opinion Term Extraction for Hotel Reviews using Transfer Learning and Auxiliary Labels
• PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data
• SkillSpan: Hard and Soft Skill Extraction from English Job Postings
• Focusing on Potential Named Entities During Active Label Acquisition
• Context-Aware Classification of Legal Document Pages
• verBERT: Automating Brazilian Case Law Document Multi-label Categorization Using BERT
• Improving Portuguese Semantic Role Labeling with Transformers and Transfer Learning
• A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets
• Predicting Legal Proceedings Status: Approaches Based on Sequential Text Data
• Optimizing small BERTs trained for German NER
• Dynamic Named Entity Recognition
• De-identification of Privacy-related Entities in Job Postings
• UoB at SemEval-2021 Task 5: Extending Pre-Trained Language Models to Include Task and Domain-Specific Information for Toxic Span Prediction
• BERT got a Date: Introducing Transformers to Temporal Tagging
• Neural Coreference Resolution for Arabic
• TMR: Evaluating NER Recall on Tough Mentions
• UIUC_BioNLP at SemEval-2021 Task 11: A Cascade of Neural Models for Structuring Scholarly NLP Contributions