Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns

Brian DuSell, David Chiang

Attention, specifically scaled dot-product attention, has proven effective for natural language, but it does not have a mechanism for handling hierarchical patterns of arbitrary ne…