Deep Networks with Internal Selective Attention through Feedback Connections
arXiv:1407.3068
Abstract
Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNets feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model.
13 pages, 3 figures
References in corpus (4)
Cited by in corpus (14)
- Striving for Simplicity: The All Convolutional Net
- Towards Deep Neural Network Architectures Robust to Adversarial Examples
- Learning Activation Functions to Improve Deep Neural Networks
- Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
- Dual Attention Networks for Multimodal Reasoning and Matching
- Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
- Hierarchical Attentive Recurrent Tracking
- Transfer entropy-based feedback improves performance in artificial neural networks
- Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
- Exploring Different Dimensions of Attention for Uncertainty Detection
- Hierarchical Spatial Transformer Network
- Classification of Radiology Reports Using Neural Attention Models
- Learning with Rethinking: Recurrently Improving Convolutional Neural Networks through Feedback
- CNNs are Globally Optimal Given Multi-Layer Support