A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land?
arXiv:2004.12126
Abstract
Fine-tuning pretrained model has achieved promising performance on standard NER benchmarks. Generally, these benchmarks are blessed with strong name regularity, high mention coverage and sufficient context diversity. Unfortunately, when scaling NER to open situations, these advantages may no longer exist. And therefore it raises a critical question of whether previous creditable approaches can still work well when facing these challenges. As there is no currently available dataset to investigate this problem, this paper proposes to conduct randomization test on standard benchmarks. Specifically, we erase name regularity, mention coverage and context diversity respectively from the benchmarks, in order to explore their impact on the generalization ability of models. To further verify our conclusions, we also construct a new open NER dataset that focuses on entity types with weaker name regularity and lower mention coverage to verify our conclusion. From both randomization test and empirical experiments, we draw the conclusions that 1) name regularity is critical for the models to generalize to unseen mentions; 2) high mention coverage may undermine the model generalization ability and 3) context patterns may not require enormous data to capture when using pretrained encoders.
arXiv commentAccepted to EMNLP2020
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Understanding deep learning requires rethinking generalization
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- A Survey on Recent Advances in Named Entity Recognition from Deep Learning models
- Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
- Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection
- Multi-Perspective Context Matching for Machine Comprehension
- Multi-Grained Named Entity Recognition
- A Unified MRC Framework for Named Entity Recognition
- Entity-Relation Extraction as Multi-Turn Question Answering
- Sequence-to-Nuggets: Nested Entity Mention Detection via Anchor-Region Networks
- Low-Resource Name Tagging Learned with Weakly Labeled Data