Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks

Abstract

Recognizing arbitrary multi-character text in unconstrained natural photographs is a hard problem. In this paper, we address an equally hard sub-problem in this domain viz. recognizing arbitrary multi-digit numbers from Street View imagery. Traditional approaches to solve this problem typically separate out the localization, segmentation, and recognition steps. In this paper we propose a unified approach that integrates these three steps via the use of a deep convolutional neural network that operates directly on the image pixels. We employ the DistBelief implementation of deep neural networks in order to train large, distributed neural networks on high quality images. We find that the performance of this approach increases with the depth of the convolutional network, with the best performance occurring in the deepest architecture we trained, with eleven hidden layers. We evaluate this approach on the publicly available SVHN dataset and achieve over $96\%$ accuracy in recognizing complete street numbers. We show that on a per-digit recognition task, we improve upon the state-of-the-art, achieving $97.84\%$ accuracy. We also evaluate this approach on an even more challenging dataset generated from Street View imagery containing several tens of millions of street number annotations and achieve over $90\%$ accuracy. To further explore the applicability of the proposed system to broader text recognition tasks, we apply it to synthetic distorted text from reCAPTCHA. reCAPTCHA is one of the most secure reverse turing tests that uses distorted text to distinguish humans from bots. We report a $99.8\%$ accuracy on the hardest category of reCAPTCHA. Our evaluations on both tasks indicate that at specific operating thresholds, the performance of the proposed system is comparable to, and in some cases exceeds, that of human operators.

Cited by in corpus (70)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
• Predictive Uncertainty Estimation via Prior Networks
• Recent Advances in Convolutional Neural Networks
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Synthetic Data for Text Localisation in Natural Images
• Motivating the Rules of the Game for Adversarial Example Research
• Self-Supervised Generalisation with Meta Auxiliary Learning
• Flowing ConvNets for Human Pose Estimation in Videos
• Recursive Recurrent Nets with Attention Modeling for OCR in the Wild
• New SAR target recognition based on YOLO and very deep multi-canonical correlation analysis
• Spatial Transformer Networks
• Gotta CAPTCHA 'Em All: A Survey of Twenty years of the Human-or-Computer Dilemma
• Universal approximations of permutation invariant/equivariant functions by deep neural networks
• Extensible Structure-Informed Prediction of Formation Energy with Improved Accuracy and Usability employing Neural Networks
• Assessing four Neural Networks on Handwritten Digit Recognition Dataset (MNIST)
• Competitive Multi-scale Convolution
• On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models
• Exploiting Local Structures with the Kronecker Layer in Convolutional Networks
• A Survey of Deep Learning Approaches for OCR and Document Understanding
• Player Identification in Hockey Broadcast Videos
• Regularizing Deep Networks with Semantic Data Augmentation
• How useful is photo-realistic rendering for visual learning?
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Fast and Robust Multiple ColorChecker Detection using Deep Convolutional Neural Networks
• A CNN Based Scene Chinese Text Recognition Algorithm With Synthetic Data Engine
• Chargrid: Towards Understanding 2D Documents
• Can Single Neurons Solve MNIST? The Computational Power of Biological Dendritic Trees
• DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)
• Revisiting Spatial Invariance with Low-Rank Local Connectivity
• A CADe System for Gliomas in Brain MRI using Convolutional Neural Networks
• Do biological constraints impair dendritic computation?
• FOTS: Fast Oriented Text Spotting with a Unified Network
• PTEENet: Post-Trained Early-Exit Neural Networks Augmentation for Inference Cost Optimization
• Large Scale Business Discovery from Street Level Imagery
• Dynamic Capacity Networks
• Residual Squeeze VGG16
• Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks
• Sequential Graph Convolutional Network for Active Learning
• Learning Fine-grained Features via a CNN Tree for Large-scale Classification
• Representational Capacity of Deep Neural Networks -- A Computing Study
• Structural Similarities Between Language Models and Neural Response Measurements
• Shredder: Learning Noise Distributions to Protect Inference Privacy
• Cascaded Subpatch Networks for Effective CNNs
• Curriculum By Smoothing
• Designing Adaptive Neural Networks for Energy-Constrained Image Classification
• Tagger: Deep Unsupervised Perceptual Grouping
• Prior Networks for Detection of Adversarial Attacks
• R-Clustering for Egocentric Video Segmentation
• Uniform Priors for Data-Efficient Transfer
• A Multiclass Multiple Instance Learning Method with Exact Likelihood
• On the Number of Linear Functions Composing Deep Neural Network: Towards a Refined Definition of Neural Networks Complexity
• Inductive Visual Localisation: Factorised Training for Superior Generalisation
• An optimized system to solve text-based CAPTCHA
• Where to Look: A Unified Attention Model for Visual Recognition with Reinforcement Learning
• Towards the Characterization of Representations Learned via Capsule-based Network Architectures
• On Study of the Binarized Deep Neural Network for Image Classification
• Comparison of Neuronal Attention Models
• Content-based similar document image retrieval using fusion of CNN features
• Attack Type Agnostic Perceptual Enhancement of Adversarial Images
• Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision
• Estimating Bicycle Route Attractivity from Image Data
• Architecture Matters: Investigating the Influence of Differential Privacy on Neural Network Design
• Unsupervised Foveal Vision Neural Networks with Top-Down Attention
• CNNs are Globally Optimal Given Multi-Layer Support
• Deep Anchored Convolutional Neural Networks
• Conditionally Learn to Pay Attention for Sequential Visual Task
• QuickBrowser: A Unified Model to Detect and Read Simple Object in Real-time