Kronecker Attention Networks
arXiv:2007.08442 · doi:10.1145/3394486.3403065
Abstract
Attention operators have been applied on both 1-D data like texts and higher-order data such as images and videos. Use of attention operators on high-order data requires flattening of the spatial or spatial-temporal dimensions into a vector, which is assumed to follow a multivariate normal distribution. This not only incurs excessive requirements on computational resources, but also fails to preserve structures in data. In this work, we propose to avoid flattening by assuming the data follow matrix-variate normal distributions. Based on this new view, we develop Kronecker attention operators (KAOs) that operate on high-order tensor data directly. More importantly, the proposed KAOs lead to dramatic reductions in computational resources. Experimental results show that our methods reduce the amount of required computational resources by a factor of hundreds, with larger factors for higher-dimensional and higher-order data. Results also show that networks with KAOs outperform models without attention, while achieving competitive performance as those with original attention operators.
9 pages, KDD2020
References in corpus (15)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Neural Machine Translation by Jointly Learning to Align and Translate
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- TensorFlow: A system for large-scale machine learning
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- ParseNet: Looking Wider to See Better
- Large-Scale Learnable Graph Convolutional Networks
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding
- Smoothed Dilated Convolutions for Improved Dense Prediction
- Transposable regularized covariance models with an application to missing data imputation
- ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
- Spatial Variational Auto-Encoding via Matrix-Variate Normal Distributions