Latin BERT: A Contextual Language Model for Classical Philology

Abstract

We present Latin BERT, a contextual language model for the Latin language, trained on 642.7 million words from a variety of sources spanning the Classical era to the 21st century. In a series of case studies, we illustrate the affordances of this language-specific model both for work in natural language processing for Latin and in using computational methods for traditional scholarship: we show that Latin BERT achieves a new state of the art for part-of-speech tagging on all three Universal Dependency datasets for Latin and can be used for predicting missing text (including critical emendations); we create a new dataset for assessing word sense disambiguation for Latin and demonstrate that Latin BERT outperforms static word embeddings; and we show that it can be used for semantically-informed search by querying contextual nearest neighbors. We publicly release trained models to help drive future work in this space.

References in corpus (11)

• Massively Multilingual Word Embeddings
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• Understanding the Behaviors of BERT in Ranking
• What the [MASK]? Making Sense of Language-Specific BERT Models
• FlauBERT: Unsupervised Language Model Pre-training for French
• BERT Rediscovers the Classical NLP Pipeline
• Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing
• GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge
• UDPipe at EvaLatin 2020: Contextualized Embeddings and Treebank Embeddings
• Restoring ancient text using deep learning: a case study on Greek epigraphy
• On the Feasibility of Automated Detection of Allusive Text Reuse

Cited by in corpus (6)

• Named Entity Recognition and Classification on Historical Documents: A Survey
• A Survey on Contextualised Semantic Shift Detection
• Statistically significant detection of semantic shifts using contextual word embeddings
• HUE: Pretrained Model and Dataset for Understanding Hanja Documents of Ancient Korea
• BERT in Plutarch's Shadows
• Latin writing styles analysis with Machine Learning: New approach to old questions