Multiple Object Recognition with Visual Attention

Abstract

We present an attention-based model for recognizing multiple objects in images. The proposed model is a deep recurrent neural network trained with reinforcement learning to attend to the most relevant regions of the input image. We show that the model learns to both localize and recognize multiple objects despite being given only class labels during training. We evaluate the model on the challenging task of transcribing house number sequences from Google Street View images and show that it is both more accurate than the state-of-the-art convolutional networks and uses fewer parameters and less computation.

References in corpus (1)

• Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition

Cited by in corpus (178)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• TensorFlow: A system for large-scale machine learning
• A Survey of Deep Learning-based Object Detection
• DRAW: A Recurrent Neural Network For Image Generation
• Deep Reinforcement Learning: An Overview
• Action Recognition using Visual Attention
• Concept Whitening for Interpretable Image Recognition
• Improved training of end-to-end attention models for speech recognition
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Image Captioning with Semantic Attention
• Learn to Combine Modalities in Multimodal Deep Learning
• Attention to Scale: Scale-aware Semantic Image Segmentation
• Learning to detect chest radiographs containing lung nodules using visual attention networks
• Semantic-Aware Scene Recognition
• Fully Convolutional Attention Networks for Fine-Grained Recognition
• Attention for Fine-Grained Categorization
• Reinforcement Learning Neural Turing Machines - Revised
• Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
• Review: Deep Learning in Electron Microscopy
• Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
• ChestNet: A Deep Neural Network for Classification of Thoracic Diseases on Chest Radiography
• Deep Reinforcement Learning for Visual Object Tracking in Videos
• GRAM: Graph-based Attention Model for Healthcare Representation Learning
• HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Dynamic Routing Between Capsules
• On the Origin of Deep Learning
• Hierarchical Object Detection with Deep Reinforcement Learning
• Tree-Structured Reinforcement Learning for Sequential Object Localization
• Attention-Aware Compositional Network for Person Re-identification
• Neural Reverse Engineering of Stripped Binaries using Augmented Control Flow Graphs
• Attentive Recurrent Comparators
• Learning Multi-Attention Context Graph for Group-Based Re-Identification
• Attention Correctness in Neural Image Captioning
• Pose-conditioned Spatio-Temporal Attention for Human Action Recognition
• Bilinear CNNs for Fine-grained Visual Recognition
• Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents
• Improving Face-Based Age Estimation with Attention-Based Dynamic Patch Fusion
• Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classification
• A Review of 40 Years of Cognitive Architecture Research: Core Cognitive Abilities and Practical Applications
• Multi-label Image Recognition by Recurrently Discovering Attentional Regions
• First Step toward Model-Free, Anonymous Object Tracking with Recurrent Neural Networks
• Fine-grained pose prediction, normalization, and recognition
• On the Binding Problem in Artificial Neural Networks
• Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition
• Video Storytelling: Textual Summaries for Events
• Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification
• Attention-gating for improved radio galaxy classification
• An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
• End-to-end Learning of Action Detection from Frame Glimpses in Videos
• Relation-Aware Global Attention for Person Re-identification
• Recurrent Spatial Transformer Networks
• Saccader: Improving Accuracy of Hard Attention Models for Vision
• Learning what to look in chest X-rays with a recurrent visual attention model
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks
• Simple and Lightweight Human Pose Estimation
• 3G structure for image caption generation
• It Takes Two to Tango: Towards Theory of AI's Mind
• Exploiting Local Structures with the Kronecker Layer in Convolutional Networks
• Joint Attention in Driver-Pedestrian Interaction: from Theory to Practice
• High Speed Human Action Recognition using a Photonic Reservoir Computer
• Learning to predict where to look in interactive environments using deep recurrent q-learning
• Towards Interpretable Reinforcement Learning Using Attention Augmented Agents
• Stacked Semantic-Guided Attention Model for Fine-Grained Zero-Shot Learning
• Attention Based Glaucoma Detection: A Large-scale Database and CNN Model
• Spatially Adaptive Computation Time for Residual Networks
• Learning efficient haptic shape exploration with a rigid tactile sensor array
• Emergence of foveal image sampling from learning to attend in visual scenes
• Agile Amulet: Real-Time Salient Object Detection with Contextual Attention
• Multi-Agent Game Abstraction via Graph Attention Neural Network
• Detecting events and key actors in multi-person videos
• Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks
• Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification
• Cardiac Segmentation on CT Images through Shape-Aware Contour Attentions
• Dynamic Computational Time for Visual Attention
• Black-Box Policy Search with Probabilistic Programs
• Learning Blended, Precise Semantic Program Embeddings
• Fast Transient Simulation of High-Speed Channels Using Recurrent Neural Network
• Multi-Label Image Classification with Contrastive Learning
• Grounding Referring Expressions in Images by Variational Context
• Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
• Dynamic Zoom-in Network for Fast Object Detection in Large Images
• ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking
• Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition
• Visual Learning of Arithmetic Operations
• Folksonomication: Predicting Tags for Movies from Plot Synopses Using Emotion Flow Encoded Neural Network
• Pack and Detect: Fast Object Detection in Videos Using Region-of-Interest Packing
• Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Video
• Saliency-based Sequential Image Attention with Multiset Prediction
• Active Perception using Light Curtains for Autonomous Driving
• Hard-Attention for Scalable Image Classification
• Quantum Tensor Network in Machine Learning: An Application to Tiny Object Classification
• Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network
• Improving Generalization of Transfer Learning Across Domains Using Spatio-Temporal Features in Autonomous Driving
• Dynamic Capacity Networks
• Why Pay More When You Can Pay Less: A Joint Learning Framework for Active Feature Acquisition and Classification
• Hierarchical Question Answering for Long Documents
• Image Captioning Based on a Hierarchical Attention Mechanism and Policy Gradient Optimization
• Improving Deep Image Clustering With Spatial Transformer Layers
• Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
• Achievements and Challenges in Explaining Deep Learning based Computer-Aided Diagnosis Systems
• Fourier Policy Gradients
• CAR-Net: Clairvoyant Attentive Recurrent Network
• A study of latent monotonic attention variants
• Object Based Attention Through Internal Gating
• Delving Deeper into the Decoder for Video Captioning
• Hide-and-Seek: A Template for Explainable AI
• Active Object Localization in Visual Situations
• Attend in groups: a weakly-supervised deep learning framework for learning from web data
• Action-Driven Object Detection with Top-Down Visual Attentions
• LS-Tree: Model Interpretation When the Data Are Linguistic
• Person Re-identification Using Visual Attention
• Hierarchical Multi-scale Attention Networks for Action Recognition
• Video Object Segmentation with Joint Re-identification and Attention-Aware Mask Propagation
• Do Autonomous Agents Benefit from Hearing?
• Image Captioning with Object Detection and Localization
• A Biologically Inspired Visual Working Memory for Deep Networks
• Learning to Search in Long Documents Using Document Structure
• An Attention-Driven Approach of No-Reference Image Quality Assessment
• Spatial-Aware Non-Local Attention for Fashion Landmark Detection
• Long Short-Term Memory Spatial Transformer Network
• Rapid Visual Categorization is not Guided by Early Salience-Based Selection
• Dynamic Filtering with Large Sampling Field for ConvNets
• A Deep Decoder Structure Based on WordEmbedding Regression for An Encoder-Decoder Based Model for Image Captioning
• Attend To Count: Crowd Counting with Adaptive Capacity Multi-scale CNNs
• Locally Smoothed Neural Networks
• Sentiment Classification with Word Attention based on Weakly Supervised Learning with a Convolutional Neural Network
• AutoScaler: Scale-Attention Networks for Visual Correspondence
• Putting visual object recognition in context
• End-to-End Speaker Height and age estimation using Attention Mechanism with LSTM-RNN
• CRAM: Clued Recurrent Attention Model
• Attention-based Transfer Learning for Brain-computer Interface
• Neural Machine Translation: A Review and Survey
• Symmetries and control in generative neural nets
• Unsupervised Deep Metric Learning with Transformed Attention Consistency and Contrastive Clustering Loss
• Fine-Grained Object Detection over Scientific Document Images with Region Embeddings
• Multi-Task Spatiotemporal Neural Networks for Structured Surface Reconstruction
• Extracting Contact and Motion from Manipulation Videos
• Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification
• ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and Interpolation
• Cell Selection with Deep Reinforcement Learning in Sparse Mobile Crowdsensing
• Gradient Importance Learning for Incomplete Observations
• Pre-training Attention Mechanisms
• VERAM: View-Enhanced Recurrent Attention Model for 3D Shape Classification
• Learning a Reinforced Agent for Flexible Exposure Bracketing Selection
• Comparison of Neuronal Attention Models
• Interest-Related Item Similarity Model Based on Multimodal Data for Top-N Recommendation
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• Lift-the-flap: what, where and when for context reasoning
• Selective Particle Attention: Visual Feature-Based Attention in Deep Reinforcement Learning
• Latent Variable Algorithms for Multimodal Learning and Sensor Fusion
• A Controller-Recognizer Framework: How necessary is recognition for control?
• Attention Driven Person Re-identification
• Real Time Incremental Foveal Texture Mapping for Autonomous Vehicles
• Deep Learning Based Decision Support for Medicine -- A Case Study on Skin Cancer Diagnosis
• ALCNN: Attention-based Model for Fine-grained Demand Inference of Dock-less Shared Bike in New Cities
• Visual Attention in Imaginative Agents
• Cost-Aware Fine-Grained Recognition for IoTs Based on Sequential Fixations
• Sidekick Policy Learning for Active Visual Exploration
• Building Intelligent Autonomous Navigation Agents
• A Reinforcement Learning Approach for Sequential Spatial Transformer Networks
• Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition
• Recurrent Existence Determination Through Policy Optimization
• Video-Based Convolutional Attention for Person Re-Identification
• Improving Automated Visual Fault Detection by Combining a Biologically Plausible Model of Visual Attention with Deep Learning
• Understanding Information Processing in Human Brain by Interpreting Machine Learning Models
• Recurrent Soft Attention Model for Common Object Recognition
• Applying deep learning to teleseismic phase detection and picking: PcP and PKiKP cases
• A Deeper Look at the Unsupervised Learning of Disentangled Representations in $\beta$-VAE from the Perspective of Core Object Recognition
• Recognizing Part Attributes with Insufficient Data
• Class Correlation affects Single Object Localization using Pre-trained ConvNets
• CHEER: Rich Model Helps Poor Model via Knowledge Infusion
• Conditionally Learn to Pay Attention for Sequential Visual Task
• Active Object Localization with Deep Reinforcement Learning
• Implications of Human Irrationality for Reinforcement Learning
• A backward pass through a CNN using a generative model of its activations
• DAVID: Dual-Attentional Video Deblurring