Tree-structured Attention with Hierarchical Accumulation
arXiv:2002.08046
Abstract
Incorporating hierarchical structures like constituency trees has been shown to be effective for various natural language processing (NLP) tasks. However, it is evident that state-of-the-art (SOTA) sequence-based models like the Transformer struggle to encode such structures inherently. On the other hand, dedicated models like the Tree-LSTM, while explicitly modeling hierarchical structures, do not perform as efficiently as the Transformer. In this paper, we attempt to bridge this gap with "Hierarchical Accumulation" to encode parse tree structures into self-attention at constant time complexity. Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT'14 English-German translation task. It also yields improvements over Transformer and Tree-LSTM on three text classification tasks. We further demonstrate that using hierarchical priors can compensate for data shortage, and that our model prefers phrase-level attentions over token-level attentions.
ICLR 2020
References in corpus (4)
Cited by in corpus (9)
- GraphCodeBERT: Pre-training Code Representations with Data Flow
- SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
- Code Prediction by Feeding Trees to Transformers
- GN-Transformer: Fusing Sequence and Graph Representation for Improved Code Summarization
- Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees
- Syntax-Enhanced Pre-trained Model
- Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization
- Do We Really Need That Many Parameters In Transformer For Extractive Summarization? Discourse Can Help !
- Resurrecting Submodularity for Neural Text Generation