Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding

Abstract

Attention-based models have shown significant improvement over traditional algorithms in several NLP tasks. The Transformer, for instance, is an illustrative example that generates abstract representations of tokens inputted to an encoder based on their relationships to all tokens in a sequence. Recent studies have shown that although such models are capable of learning syntactic features purely by seeing examples, explicitly feeding this information to deep learning models can significantly enhance their performance. Leveraging syntactic information like part of speech (POS) may be particularly beneficial in limited training data settings for complex models such as the Transformer. We show that the syntax-infused Transformer with multiple features achieves an improvement of 0.7 BLEU when trained on the full WMT 14 English to German translation dataset and a maximum improvement of 1.99 BLEU points when trained on a fraction of the dataset. In addition, we find that the incorporation of syntax into BERT fine-tuning outperforms baseline on a number of downstream tasks from the GLUE benchmark.

References in corpus (6)

• Semi-Supervised Classification with Graph Convolutional Networks
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Distance-based Self-Attention Network for Natural Language Inference
• Multi-Head Attention with Disagreement Regularization
• Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning
• Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention

Cited by in corpus (8)

• MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
• Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale
• Causality in Requirements Artifacts: Prevalence, Detection, and Impact
• Semantics-aware Attention Improves Neural Machine Translation
• Semi-Supervised Few-Shot Intent Classification and Slot Filling
• Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees
• Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling
• What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Study