Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding
arXiv:1908.06006
Abstract
The Hierarchical Attention Network (HAN) has made great strides, but it suffers a major limitation: at level 1, each sentence is encoded in complete isolation. In this work, we propose and compare several modifications of HAN in which the sentence encoder is able to make context-aware attentional decisions (CAHAN). Furthermore, we propose a bidirectional document encoder that processes the document forwards and backwards, using the preceding and following sentences as context. Experiments on three large-scale sentiment and topic classification datasets show that the bidirectional version of CAHAN outperforms HAN everywhere, with only a modest increase in computation time. While results are promising, we expect the superiority of CAHAN to be even more evident on tasks requiring a deeper understanding of the input documents, such as abstractive summarization. Code is publicly available.
References in corpus (7)
- Efficient Estimation of Word Representations in Vector Space
- A Structured Self-attentive Sentence Embedding
- A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
- Three Factors Influencing Minima in SGD
- Does Neural Machine Translation Benefit from Larger Context?
- Energy-based Self-attentive Learning of Abstractive Communities for Spoken Language Understanding
- Notes on Deep Learning for NLP