Full Page Handwriting Recognition via Image to Sequence Extraction

Abstract

We present a Neural Network based Handwritten Text Recognition (HTR) model architecture that can be trained to recognize full pages of handwritten or printed text without image segmentation. Being based on Image to Sequence architecture, it can extract text present in an image and then sequence it correctly without imposing any constraints regarding orientation, layout and size of text and non-text. Further, it can also be trained to generate auxiliary markup related to formatting, layout and content. We use character level vocabulary, thereby enabling language and terminology of any subject. The model achieves a new state-of-art in paragraph level recognition on the IAM dataset. When evaluated on scans of real world handwritten free form test answers - beset with curved and slanted lines, drawings, tables, math, chemistry and other symbols - it performs better than all commercially available HTR cloud APIs. It is deployed in production as part of a commercial web application.

Appeared in ICDAR 2021

References in corpus (5)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Sequence to Sequence Learning with Neural Networks
• Longformer: The Long-Document Transformer
• Pointer Sentinel Mixture Models
• Decoupled Attention Network for Text Recognition

Cited by in corpus (3)

• DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition
• Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognition
• Faster DAN: Multi-target Queries with Document Positional Encoding for End-to-end Handwritten Document Recognition