Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling

Abstract

Recurrent neural networks (RNN), convolutional neural networks (CNN) and self-attention networks (SAN) are commonly used to produce context-aware representations. RNN can capture long-range dependency but is hard to parallelize and not time-efficient. CNN focuses on local dependency but does not perform well on some tasks. SAN can model both such dependencies via highly parallelizable computation, but memory requirement grows rapidly in line with sequence length. In this paper, we propose a model, called "bi-directional block self-attention network (Bi-BloSAN)", for RNN/CNN-free sequence encoding. It requires as little memory as RNN but with all the merits of SAN. Bi-BloSAN splits the entire sequence into blocks, and applies an intra-block SAN to each block for modeling local context, then applies an inter-block SAN to the outputs for all blocks to capture long-range dependency. Thus, each SAN only needs to process a short sequence, and only a small amount of memory is required. Additionally, we use feature-level attention to handle the variation of contexts around the same word, and use forward/backward masks to encode temporal order information. On nine benchmark datasets for different NLP tasks, Bi-BloSAN achieves or improves upon state-of-the-art accuracy, and shows better efficiency-memory trade-off than existing RNN/CNN/SAN.

arXiv comment18 pages, 7 figures; Accepted in ICLR-18

References in corpus (19)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• ADADELTA: An Adaptive Learning Rate Method
• Convolutional Sequence to Sequence Learning
• A Structured Self-attentive Sentence Embedding
• Language Modeling with Gated Convolutional Networks
• Skip-Thought Vectors
• End-To-End Memory Networks
• Neural Machine Translation in Linear Time
• Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
• Multi-Perspective Context Matching for Machine Comprehension
• Structured Attention Networks
• Linguistically Regularized LSTMs for Sentiment Classification
• Self-Adaptive Hierarchical Sentence Model
• Reinforced Mnemonic Reader for Machine Reading Comprehension
• A Fast Unified Model for Parsing and Sentence Understanding
• Bidirectional Tree-Structured LSTM with Head Lexicalization
• Ruminating Reader: Reasoning with Gated Multi-Hop Attention

Cited by in corpus (25)

• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• A Survey on Text Classification: From Shallow to Deep Learning
• Sparse Sinkhorn Attention
• Text Classification using Capsules
• Dialogue Act Classification with Context-Aware Self-Attention
• BP-Transformer: Modelling Long-Range Context via Binary Partitioning
• Describing a Knowledge Base
• Dynamic Self-Attention : Computing Attention over Words Dynamically for Sentence Embedding
• ChaLearn Looking at People: IsoGD and ConGD Large-scale RGB-D Gesture Recognition
• Temporal Self-Attention Network for Medical Concept Embedding
• One model Packs Thousands of Items with Recurrent Conditional Query Learning
• ETC: Encoding Long and Structured Inputs in Transformers
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• Conditional Self-Attention for Query-based Summarization
• On the Importance of Attention in Meta-Learning for Few-Shot Text Classification
• Multi-Granularity Self-Attention for Neural Machine Translation
• Label-guided Learning for Text Classification
• Self-Attention Enhanced Patient Journey Understanding in Healthcare System
• Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together
• FastFusionNet: New State-of-the-Art for DAWNBench SQuAD
• "Bilingual Expert" Can Find Translation Errors
• On the Importance of Local Information in Transformer Based Models
• Improving Multi-Head Attention with Capsule Networks
• Cascaded Semantic and Positional Self-Attention Network for Document Classification
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes