Rosetta: Large scale system for text detection and recognition in images

Abstract

In this paper we present a deployed, scalable optical character recognition (OCR) system, which we call Rosetta, designed to process images uploaded daily at Facebook scale. Sharing of image content has become one of the primary ways to communicate information among internet users within social networks such as Facebook and Instagram, and the understanding of such media, including its textual information, is of paramount importance to facilitate search and recommendation applications. We present modeling techniques for efficient detection and recognition of text in images and describe Rosetta's system architecture. We perform extensive evaluation of presented technologies, explain useful practical approaches to build an OCR system at scale, and provide insightful intuitions as to why and how certain components work based on the lessons learnt during the development and deployment of the system.

Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) 2018, London, United Kingdom

References in corpus (7)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
• COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images
• EAST: An Efficient and Accurate Scene Text Detector
• Scene Text Recognition with Sliding Convolutional Character Models
• Reading Scene Text with Attention Convolutional Sequence Modeling

Cited by in corpus (47)

• Scene Text Detection and Recognition: The Deep Learning Era
• Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark
• Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
• What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis
• Towards Image-based Automatic Meter Reading in Unconstrained Scenarios: A Robust and Efficient Approach
• Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
• CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues
• Image-based Automatic Dial Meter Reading in Unconstrained Scenarios
• Reinterpreting CTC training as iterative fitting
• Exploring Sparse Spatial Relation in Graph Inference for Text-Based VQA
• Text Detection and Recognition in the Wild: A Review
• Human-Adversarial Visual Question Answering
• First-Generation Inference Accelerator Deployment at Facebook
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
• Leveraging Model Fusion for Improved License Plate Recognition
• Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
• Structured Multimodal Attentions for TextVQA
• MMOCR: A Comprehensive Toolbox for Text Detection, Recognition and Understanding
• Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text
• TRIE: End-to-End Text Reading and Information Extraction for Document Understanding
• UTRNet: High-Resolution Urdu Text Recognition In Printed Documents
• Offline Handwritten Chinese Text Recognition with Convolutional Neural Networks
• Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering
• Advancing Multinational License Plate Recognition Through Synthetic and Real Data Fusion: A Comprehensive Evaluation
• SCATTER: Selective Context Attentional Scene Text Recognizer
• You Only Recognize Once: Towards Fast Video Text Spotting
• Revisiting Classification Perspective on Scene Text Recognition
• STRIDE : Scene Text Recognition In-Device
• Towards Accurate Text-based Image Captioning with Content Diversity Exploration
• Improving Long Handwritten Text Line Recognition with Convolutional Multi-way Associative Memory
• Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems
• 3D Rendering Framework for Data Augmentation in Optical Character Recognition
• DM${}^2$S${}^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
• Digital Peter: Dataset, Competition and Handwriting Recognition Methods
• EASTER: Efficient and Scalable Text Recognizer
• Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models
• ReADS: A Rectified Attentional Double Supervised Network for Scene Text Recognition
• Deep Multi-Modal Sets
• Robust Handwriting Recognition with Limited and Noisy Data
• Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling
• Utilizing Resource-Rich Language Datasets for End-to-End Scene Text Recognition in Resource-Poor Languages
• A Machine Learning Framework for Data Ingestion in Document Images
• Graph Relation Transformer: Incorporating pairwise object features into the Transformer architecture
• Assessing Individual and Community Vulnerability to Fake News in Social Networks
• Using Human Psychophysics to Evaluate Generalization in Scene Text Recognition Models