Deep Networks with Internal Selective Attention through Feedback Connections

Abstract

Traditional convolutional neural networks (CNN) are stationary and feedforward. They neither change their parameters during evaluation nor use feedback from higher to lower layers. Real brains, however, do. So does our Deep Attention Selective Network (dasNet) architecture. DasNets feedback structure can dynamically alter its convolutional filter sensitivities during classification. It harnesses the power of sequential processing to improve classification performance, by allowing the network to iteratively focus its internal attention on some of its convolutional filters. Feedback is trained through direct policy search in a huge million-dimensional parameter space, through scalable natural evolution strategies (SNES). On the CIFAR-10 and CIFAR-100 datasets, dasNet outperforms the previous state-of-the-art model.

13 pages, 3 figures

References in corpus (5)

• Deep Learning in Neural Networks: An Overview
• Improving neural networks by preventing co-adaptation of feature detectors
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Visualizing and Understanding Convolutional Networks
• Multi-column Deep Neural Networks for Image Classification

Cited by in corpus (38)

• Striving for Simplicity: The All Convolutional Net
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Large-Margin Softmax Loss for Convolutional Neural Networks
• Learning Activation Functions to Improve Deep Neural Networks
• Residual Attention Network for Image Classification
• Attention Enriched Deep Learning Model for Breast Tumor Segmentation in Ultrasound Images
• Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• Spatial Transformer Networks
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Deep Predictive Coding Network for Object Recognition
• Dual Attention Networks for Multimodal Reasoning and Matching
• HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition
• Sketch-a-Net that Beats Humans
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
• Granger-causal Attentive Mixtures of Experts: Learning Important Features with Neural Networks
• Hierarchical Attentive Recurrent Tracking
• Lexicon Integrated CNN Models with Attention for Sentiment Analysis
• Dynamic Computational Time for Visual Attention
• Visual Question Answering with Memory-Augmented Networks
• Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
• Pixel-aware Deep Function-mixture Network for Spectral Super-Resolution
• Transfer entropy-based feedback improves performance in artificial neural networks
• Object Based Attention Through Internal Gating
• Feature-based Attention in Convolutional Neural Networks
• The Costs and Benefits of Goal-Directed Attention in Deep Convolutional Neural Networks
• Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
• Exploring Different Dimensions of Attention for Uncertainty Detection
• Neural Network Encapsulation
• Hierarchical Spatial Transformer Network
• Visual Sensation and Perception Computational Models for Deep Learning: State of the art, Challenges and Prospects
• Classification of Radiology Reports Using Neural Attention Models
• Learning with Rethinking: Recurrently Improving Convolutional Neural Networks through Feedback
• Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition
• Attention Incorporate Network: A network can adapt various data size
• CNNs are Globally Optimal Given Multi-Layer Support
• Solar Power Plant Detection on Multi-Spectral Satellite Imagery using Weakly-Supervised CNN with Feedback Features and m-PCNN Fusion