Multi-Zone Unit for Recurrent Neural Networks
arXiv:1911.07184
Abstract
Recurrent neural networks (RNNs) have been widely used to deal with sequence learning problems. The input-dependent transition function, which folds new observations into hidden states to sequentially construct fixed-length representations of arbitrary-length sequences, plays a critical role in RNNs. Based on single space composition, transition functions in existing RNNs often have difficulty in capturing complicated long-range dependencies. In this paper, we introduce a new Multi-zone Unit (MZU) for RNNs. The key idea is to design a transition function that is capable of modeling multiple space composition. The MZU consists of three components: zone generation, zone composition, and zone aggregation. Experimental results on multiple datasets of the character-level language modeling task and the aspect-based sentiment analysis task demonstrate the superiority of the MZU.
arXiv commentAccepted at AAAI 2020
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Variational Graph Auto-Encoders
- Regularizing and Optimizing LSTM Language Models
- Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
- Adaptive Computation Time for Recurrent Neural Networks
- Hierarchical Multiscale Recurrent Neural Networks
- Architectural Complexity Measures of Recurrent Neural Networks
- Multiplicative LSTM for sequence modelling
- Aspect Based Sentiment Analysis with Gated Convolutional Networks
- On Multiplicative Integration with Recurrent Neural Networks
- Fast-Slow Recurrent Neural Networks
- Recurrent Attention Unit
- Long Short-Term Attention
- A Novel Aspect-Guided Deep Transition Model for Aspect Based Sentiment Analysis