Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT

Abstract

We combine character-level and contextual language model representations to improve performance on Discourse Representation Structure parsing. Character representations can easily be added in a sequence-to-sequence model in either one encoder or as a fully separate encoder, with improvements that are robust to different language models, languages and data sets. For English, these improvements are larger than adding individual sources of linguistic information or adding non-contextual embeddings. A new method of analysis based on semantic tags demonstrates that the character-level representations improve performance across a subset of selected semantic phenomena.

arXiv commentEMNLP 2020 (long)

References in corpus (7)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Character-Aware Neural Language Models
• Training Tips for the Transformer Model
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Neural Semantic Parsing by Character-based Translation: Experiments with Abstract Meaning Representations
• Semantic Tagging with Deep Residual Networks
• Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension