Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN
arXiv:2106.12901
Abstract
The recurrent network architecture is a widely used model in sequence modeling, but its serial dependency hinders the computation parallelization, which makes the operation inefficient. The same problem was encountered in serial adder at the early stage of digital electronics. In this paper, we discuss the similarities between recurrent neural network (RNN) and serial adder. Inspired by carry-lookahead adder, we introduce carry-lookahead module to RNN, which makes it possible for RNN to run in parallel. Then, we design the method of parallel RNN computation, and finally Carry-lookahead RNN (CL-RNN) is proposed. CL-RNN takes advantages in parallelism and flexible receptive field. Through a comprehensive set of tests, we verify that CL-RNN can perform better than existing typical RNNs in sequence modeling tasks which are specially designed for RNNs.
References in corpus (18)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- LSTM: A Search Space Odyssey
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Distributed Representations of Sentences and Documents
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Exploring the Limits of Language Modeling
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Regularizing and Optimizing LSTM Language Models
- Quasi-Recurrent Neural Networks
- Neural Machine Translation in Linear Time
- How to Construct Deep Recurrent Neural Networks
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Architectural Complexity Measures of Recurrent Neural Networks
- Factorization tricks for LSTM networks
- Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
- Gated Word-Character Recurrent Language Model
- Regularizing RNNs by Stabilizing Activations