Glyce: Glyph-vectors for Chinese Character Representations

Abstract

It is intuitive that NLP tasks for logographic languages like Chinese should benefit from the use of the glyph information in those languages. However, due to the lack of rich pictographic evidence in glyphs and the weak generalization ability of standard computer vision models on character data, an effective way to utilize the glyph information remains to be found. In this paper, we address this gap by presenting Glyce, the glyph-vectors for Chinese character representations. We make three major innovations: (1) We use historical Chinese scripts (e.g., bronzeware script, seal script, traditional Chinese, etc) to enrich the pictographic evidence in characters; (2) We design CNN structures (called tianzege-CNN) tailored to Chinese character image processing; and (3) We use image-classification as an auxiliary task in a multi-task learning setup to increase the model's ability to generalize. We show that glyph-based models are able to consistently outperform word/char ID-based models in a wide range of Chinese NLP tasks. We are able to set new state-of-the-art results for a variety of Chinese NLP tasks, including tagging (NER, CWS, POS), sentence pair classification, single sentence classification tasks, dependency parsing, and semantic role labeling. For example, the proposed model achieves an F1 score of 80.6 on the OntoNotes dataset of NER, +1.5 over BERT; it achieves an almost perfect accuracy of 99.8\% on the Fudan corpus for text classification. Code found at https://github.com/ShannonAI/glyce.

Accepted by NeurIPS 2019

References in corpus (25)

• Character-Aware Neural Language Models
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Natural Language Inference over Interaction Space
• When Are Tree Structures Necessary for Deep Learning of Representations?
• Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF
• Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations
• A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
• Which Encoding is the Best for Text Classification in Chinese, English, Japanese and Korean?
• Subword Encoding in Lattice LSTM for Chinese Word Segmentation
• Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning
• Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder
• Adversarial Multi-Criteria Learning for Chinese Word Segmentation
• Neural Word Segmentation Learning for Chinese
• Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
• Learning Chinese Word Representations From Glyphs Of Characters
• Training with Exploration Improves a Greedy Stack-LSTM Parser
• Component-Enhanced Chinese Character Embeddings
• Bi-directional Attention with Agreement for Dependency Parsing
• Prior Knowledge Integration for Neural Machine Translation using Posterior Regularization
• Bag-of-Words as Target for Neural Machine Translation
• State-of-the-art Chinese Word Segmentation with Bi-LSTMs
• Glyph-aware Embedding of Chinese Characters
• Fast and Accurate Neural Word Segmentation for Chinese
• Multimodal neural pronunciation modeling for spoken languages with logographic origin

Cited by in corpus (11)

• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
• A Unified MRC Framework for Named Entity Recognition
• Universal Multimodal Representation for Language Understanding
• Dice Loss for Data-imbalanced NLP Tasks
• Language Modelling with Pixels
• Chinese Embedding via Stroke and Glyph Information: A Dual-channel View
• Using Chinese Glyphs for Named Entity Recognition
• Probing Contextualized Sentence Representations with Visual Awareness
• Query-Based Named Entity Recognition
• Accurate Word Representations with Universal Visual Guidance