Deep Structured Output Learning for Unconstrained Text Recognition

Abstract

We develop a representation suitable for the unconstrained recognition of words in natural images: the general case of no fixed lexicon and unknown length. To this end we propose a convolutional neural network (CNN) based architecture which incorporates a Conditional Random Field (CRF) graphical model, taking the whole word image as a single input. The unaries of the CRF are provided by a CNN that predicts characters at each position of the output, while higher order terms are provided by another CNN that detects the presence of N-grams. We show that this entire model (CRF, character predictor, N-gram predictor) can be jointly optimised by back-propagating the structured output loss, essentially requiring the system to perform multi-task learning, and training uses purely synthetically generated data. The resulting model is a more accurate system on standard real-world text recognition benchmarks than character prediction alone, setting a benchmark for systems that have not been trained on a particular lexicon. In addition, our model achieves state-of-the-art accuracy in lexicon-constrained scenarios, without being specifically modelled for constrained recognition. To test the generalisation of our model, we also perform experiments with random alpha-numeric strings to evaluate the method when no visual language model is applicable.

arXiv admin note: text overlap with arXiv:1406.2227

References in corpus (3)

• Improving neural networks by preventing co-adaptation of feature detectors
• Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
• End-to-End Text Recognition with Hybrid HMM Maxout Models

Cited by in corpus (49)

• Learning Transferable Visual Models From Natural Language Supervision
• Conditional Random Fields as Recurrent Neural Networks
• Recent Advances in Convolutional Neural Networks
• Scene Text Detection and Recognition: The Deep Learning Era
• Robust Scene Text Recognition with Automatic Rectification
• Flowing ConvNets for Human Pose Estimation in Videos
• Recursive Recurrent Nets with Attention Modeling for OCR in the Wild
• An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition
• Towards Accurate Scene Text Recognition with Semantic Reasoning Networks
• Reading Scene Text with Attention Convolutional Sequence Modeling
• 2D Attentional Irregular Scene Text Recognizer
• Text Recognition in the Wild: A Survey
• Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes
• Reading Scene Text in Deep Convolutional Sequences
• Scene Text Recognition from Two-Dimensional Perspective
• Enhancing Energy Minimization Framework for Scene Text Recognition with Top-Down Cues
• Scene Text Synthesis for Efficient and Effective Deep Network Training
• Edit Probability for Scene Text Recognition
• ESIR: End-to-end Scene Text Recognition via Iterative Image Rectification
• 2D-CTC for Scene Text Recognition
• Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation
• SCAN: Sliding Convolutional Attention Network for Scene Text Recognition
• NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition
• Context-aware CNNs for person head detection
• Structured Prediction Energy Networks
• Focus-Enhanced Scene Text Recognition with Deformable Convolutions
• TextScanner: Reading Characters in Order for Robust Scene Text Recognition
• Verisimilar Image Synthesis for Accurate Detection and Recognition of Texts in Scenes
• Translating Math Formula Images to LaTeX Sequences Using Deep Neural Networks with Sequence-level Training
• Recurrent Calibration Network for Irregular Text Recognition
• Adaptive Adversarial Attack on Scene Text Recognition
• Bidirectional Scene Text Recognition with a Single Decoder
• ALOHA: Auxiliary Loss Optimization for Hypothesis Augmentation
• Learning Fine-grained Features via a CNN Tree for Large-scale Classification
• LEWIS: Latent Embeddings for Word Images and their Semantics
• Separating Content from Style Using Adversarial Learning for Recognizing Text in the Wild
• Learning to Read by Spelling: Towards Unsupervised Text Recognition
• AON: Towards Arbitrarily-Oriented Text Recognition
• EASTER: Efficient and Scalable Text Recognizer
• Simultaneous Recognition of Horizontal and Vertical Text in Natural Images
• Parallel Scale-wise Attention Network for Effective Scene Text Recognition
• I2C2W: Image-to-Character-to-Word Transformers for Accurate Scene Text Recognition
• Sequence to sequence learning for unconstrained scene text recognition
• Adaptive Embedding Gate for Attention-Based Scene Text Recognition
• Scaling Matters in Deep Structured-Prediction Models
• Deep Neural Networks Regularization for Structured Output Prediction
• Boosting Scene Character Recognition by Learning Canonical Forms of Glyphs
• Writing Style Invariant Deep Learning Model for Historical Manuscripts Alignment
• Scene Text Recognition With Finer Grid Rectification