GlyphCRM: Bidirectional Encoder Representation for Chinese Character with its Glyph

Abstract

Previous works indicate that the glyph of Chinese characters contains rich semantic information and has the potential to enhance the representation of Chinese characters. The typical method to utilize the glyph features is by incorporating them into the character embedding space. Inspired by previous methods, we innovatively propose a Chinese pre-trained representation model named as GlyphCRM, which abandons the ID-based character embedding method yet solely based on sequential character images. We render each character into a binary grayscale image and design two-channel position feature maps for it. Formally, we first design a two-layer residual convolutional neural network, namely HanGlyph to generate the initial glyph representation of Chinese characters, and subsequently adopt multiple bidirectional encoder Transformer blocks as the superstructure to capture the context-sensitive information. Meanwhile, we feed the glyph features extracted from each layer of the HanGlyph module into the underlying Transformer blocks by skip-connection method to fully exploit the glyph features of Chinese characters. As the HanGlyph module can obtain a sufficient glyph representation of any Chinese character, the long-standing out-of-vocabulary problem could be effectively solved. Extensive experimental results indicate that GlyphCRM substantially outperforms the previous BERT-based state-of-the-art model on 9 fine-tuning tasks, and it has strong transferability and generalization on specialized fields and low-resource tasks. We hope this work could spark further research beyond the realms of well-established representation of Chinese texts.

11 pages, 7 figures

References in corpus (13)

• Distributed Representations of Words and Phrases and their Compositionality
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Deep Learning using Rectified Linear Units (ReLU)
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• CamemBERT: a Tasty French Language Model
• mT5: A massively multilingual pre-trained text-to-text transformer
• Large-scale Simple Question Answering with Memory Networks
• Layer Normalization
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Stacked DeBERT: All Attention in Incomplete Data for Text Classification
• IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles