AnchiBERT: A Pre-Trained Model for Ancient ChineseLanguage Understanding and Generation
arXiv:2009.11473
Abstract
Ancient Chinese is the essence of Chinese culture. There are several natural language processing tasks of ancient Chinese domain, such as ancient-modern Chinese translation, poem generation, and couplet generation. Previous studies usually use the supervised models which deeply rely on parallel data. However, it is difficult to obtain large-scale parallel data of ancient Chinese. In order to make full use of the more easily available monolingual ancient Chinese corpora, we release AnchiBERT, a pre-trained language model based on the architecture of BERT, which is trained on large-scale ancient Chinese corpora. We evaluate AnchiBERT on both language understanding and generation tasks, including poem classification, ancient-modern Chinese translation, poem generation, and couplet generation. The experimental results show that AnchiBERT outperforms BERT as well as the non-pretrained models and achieves state-of-the-art results in all cases.
arXiv comment10 pages with 3 figures
References in corpus (10)
- Efficient Estimation of Word Representations in Vector Space
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- CamemBERT: a Tasty French Language Model
- AraBERT: Transformer-based Model for Arabic Language Understanding
- BERTje: A Dutch BERT Model
- Explicit State Tracking with Semi-Supervision for Neural Dialogue Generation
- A Multi-Modal Chinese Poetry Generation Model
- Ancient-Modern Chinese Translation with a Large Training Dataset
- Rethinking Dialogue State Tracking with Reasoning