FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding
arXiv:2111.14031
Abstract
Inducing latent tree structures from sequential data is an emerging trend in the NLP research landscape today, largely popularized by recent methods such as Gumbel LSTM and Ordered Neurons (ON-LSTM). This paper proposes FASTTREES, a new general purpose neural module for fast sequence encoding. Unlike most previous works that consider recurrence to be necessary for tree induction, our work explores the notion of parallel tree induction, i.e., imbuing our model with hierarchical inductive biases in a parallelizable, non-autoregressive fashion. To this end, our proposed FASTTREES achieves competitive or superior performance to ON-LSTM on four well-established sequence modeling tasks, i.e., language modeling, logical inference, sentiment analysis and natural language inference. Moreover, we show that the FASTTREES module can be applied to enhance Transformer models, achieving performance gains on three sequence transduction tasks (machine translation, subject-verb agreement and mathematical language understanding), paving the way for modular tree induction modules. Overall, we outperform existing state-of-the-art models on logical inference tasks by +4% and mathematical language understanding by +8%.
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Regularizing and Optimizing LSTM Language Models
- A large annotated corpus for learning natural language inference
- Quasi-Recurrent Neural Networks
- Pay Less Attention with Lightweight and Dynamic Convolutions
- Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
- Learning to Compose Words into Sentences with Reinforcement Learning
- A Fast Unified Model for Parsing and Sentence Understanding
- Neural Language Modeling by Jointly Learning Syntax and Lexicon
- Attending to Mathematical Language with Transformers
- StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling