Does Chinese BERT Encode Word Structure?
arXiv:2010.07711
Abstract
Contextualized representations give significantly improved results for a wide range of NLP tasks. Much work has been dedicated to analyzing the features captured by representative models such as BERT. Existing work finds that syntactic, semantic and word sense knowledge are encoded in BERT. However, little work has investigated word features for character-based languages such as Chinese. We investigate Chinese BERT using both attention weight distribution statistics and probing tasks, finding that (1) word information is captured by BERT; (2) word-level features are mostly in the middle representation layers; (3) downstream tasks make different use of word features in BERT, with POS tagging and chunking relying the most on word features, and natural language inference relying the least on such features.
arXiv commentAccepted by COLING2020
References in corpus (5)
- ERNIE: Enhanced Representation through Knowledge Integration
- Assessing BERT's Syntactic Abilities
- Design Challenges and Misconceptions in Neural Sequence Labeling
- What do you learn from context? Probing for sentence structure in contextualized word representations
- Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF