TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding
arXiv:2003.07000
Abstract
Bidirectional Encoder Representations from Transformers (BERT) has recently achieved state-of-the-art performance on a broad range of NLP tasks including sentence classification, machine translation, and question answering. The BERT model architecture is derived primarily from the transformer. Prior to the transformer era, bidirectional Long Short-Term Memory (BLSTM) has been the dominant modeling architecture for neural machine translation and question answering. In this paper, we investigate how these two modeling techniques can be combined to create a more powerful model architecture. We propose a new architecture denoted as Transformer with BLSTM (TRANS-BLSTM) which has a BLSTM layer integrated to each transformer block, leading to a joint modeling framework for transformer and BLSTM. We show that TRANS-BLSTM models consistently lead to improvements in accuracy compared to BERT baselines in GLUE and SQuAD 1.1 experiments. Our TRANS-BLSTM model obtains an F1 score of 94.01% on the SQuAD 1.1 development dataset, which is comparable to the state-of-the-art result.
References in corpus (11)
- Distributed Representations of Words and Phrases and their Compositionality
- Distilling the Knowledge in a Neural Network
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Layer Normalization
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
- Multi-Task Deep Neural Networks for Natural Language Understanding