Towards Better Modeling Hierarchical Structure for Self-Attention with Ordered Neurons
arXiv:1909.01562
Abstract
Recent studies have shown that a hybrid of self-attention networks (SANs) and recurrent neural networks (RNNs) outperforms both individual architectures, while not much is known about why the hybrid models work. With the belief that modeling hierarchical structure is an essential complementary between SANs and RNNs, we propose to further enhance the strength of hybrid models with an advanced variant of RNNs - Ordered Neurons LSTM (ON-LSTM), which introduces a syntax-oriented inductive bias to perform tree-like composition. Experimental results on the benchmark machine translation task show that the proposed approach outperforms both individual architectures and a standard hybrid model. Further analyses on targeted linguistic evaluation and logical inference tasks demonstrate that the proposed approach indeed benefits from a better modeling of hierarchical structure.
arXiv commentEMNLP 2019
References in corpus (5)
- A Structured Self-attentive Sentence Embedding
- Convolutional Neural Networks over Tree Structures for Programming Language Processing
- What you can cram into a single vector: Probing sentence embeddings for linguistic properties
- Transformers with convolutional context for ASR
- Tree-structured composition in neural networks without tree-structured architectures