Structured Attention Networks

Abstract

Attention networks have proven to be an effective approach for embedding categorical inference within a deep neural network. However, for many tasks we may want to model richer structural dependencies without abandoning end-to-end training. In this work, we experiment with incorporating richer structural distributions, encoded using graphical models, within deep networks. We show that these structured attention networks are simple extensions of the basic attention procedure, and that they allow for extending attention beyond the standard soft-selection approach, such as attending to partial segmentations or to subtrees. We experiment with two different classes of structured attention networks: a linear-chain conditional random field and a graph-based parsing model, and describe how these models can be practically implemented as neural network layers. Experiments show that this approach is effective for incorporating structural biases, and structured attention networks outperform baseline attention models on a variety of synthetic and real tasks: tree transduction, neural machine translation, question answering, and natural language inference. We further find that models trained in this way learn interesting unsupervised hidden representations that generalize simple attention.

ICLR 2017

Cited by in corpus (31)

• Attention in Natural Language Processing
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
• Describing a Knowledge Base
• Hierarchical Transformers for Multi-Document Summarization
• Reasoning with Latent Structure Refinement for Document-Level Relation Extraction
• Structured Attentions for Visual Question Answering
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence
• Semantic Role Labeling as Dependency Parsing: Exploring Latent Tree Structures Inside Arguments
• LP-SparseMAP: Differentiable Relaxed Optimization for Sparse Structured Prediction
• Modeling Latent Sentence Structure in Neural Machine Translation
• Self-Attention with Structural Position Representations
• Speaker-Oriented Latent Structures for Dialogue-Based Relation Extraction
• Recovering Dropped Pronouns in Chinese Conversations via Modeling Their Referents
• Long Short-Term Attention
• Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together
• Unsupervised Recurrent Neural Network Grammars
• Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs
• Multiresolution Transformer Networks: Recurrence is Not Essential for Modeling Hierarchical Structure
• Unsupervised Neural Single-Document Summarization of Reviews via Learning Latent Discourse Structure and its Ranking
• Vertical-Horizontal Structured Attention for Generating Music with Chords
• Controlling Utterance Length in NMT-based Word Segmentation with Attention
• Not All Attention Is Needed: Gated Attention Network for Sequence Data
• Spatial-Temporal Deep Intention Destination Networks for Online Travel Planning
• Looking for change? Roll the Dice and demand Attention
• Neural Latent Dependency Model for Sequence Labeling
• Latent-Variable Generative Models for Data-Efficient Text Classification
• To be Closer: Learning to Link up Aspects with Opinions
• Generative latent neural models for automatic word alignment
• Multilevel Text Alignment with Cross-Document Attention
• Notes on Latent Structure Models and SPIGOT