Hierarchical Character Embeddings: Learning Phonological and Semantic Representations in Languages of Logographic Origin using Recursive Neural Networks

Abstract

Logographs (Chinese characters) have recursive structures (i.e. hierarchies of sub-units in logographs) that contain phonological and semantic information, as developmental psychology literature suggests that native speakers leverage on the structures to learn how to read. Exploiting these structures could potentially lead to better embeddings that can benefit many downstream tasks. We propose building hierarchical logograph (character) embeddings from logograph recursive structures using treeLSTM, a recursive neural network. Using recursive neural network imposes a prior on the mapping from logographs to embeddings since the network must read in the sub-units in logographs according to the order specified by the recursive structures. Based on human behavior in language learning and reading, we hypothesize that modeling logographs' structures using recursive neural network should be beneficial. To verify this claim, we consider two tasks (1) predicting logographs' Cantonese pronunciation from logographic structures and (2) language modeling. Empirical results show that the proposed hierarchical embeddings outperform baseline approaches. Diagnostic analysis suggests that hierarchical embeddings constructed using treeLSTM is less sensitive to distractors, thus is more robust, especially on complex logographs.

arXiv commentAccepted by IEEE Transactions on Audio, Speech and Language Processing. Copyright 2019 IEEE

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Character-Aware Neural Language Models
• When Are Tree Structures Necessary for Deep Learning of Representations?
• Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering
• On-the-fly Operation Batching in Dynamic Computation Graphs
• Chinese-Japanese Unsupervised Neural Machine Translation Using Sub-character Level Information
• Radical-level Ideograph Encoder for RNN-based Sentiment Analysis of Chinese and Japanese
• Neural Machine Translation of Logographic Languages Using Sub-character Level Information
• Japanese Sentiment Classification using a Tree-Structured Long Short-Term Memory with Attention

Cited by in corpus (2)

• Exploring the contextual factors affecting multimodal emotion recognition in videos
• Helpful Neighbors: Leveraging Neighbors in Geographic Feature Pronunciation