Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network

Abstract

We present an end-to-end, multimodal, fully convolutional network for extracting semantic structures from document images. We consider document semantic structure extraction as a pixel-wise segmentation task, and propose a unified model that classifies pixels based not only on their visual appearance, as in the traditional page segmentation task, but also on the content of underlying text. Moreover, we propose an efficient synthetic document generation process that we use to generate pretraining data for our network. Once the network is trained on a large set of synthetic documents, we fine-tune the network on unlabeled real documents using a semi-supervised approach. We systematically study the optimum network architecture and show that both our multimodal approach and the synthetic data pretraining significantly boost the performance.

CVPR 2017 Spotlight

Cited by in corpus (21)

• Extracting Scientific Figures with Distantly Supervised Neural Networks
• Fully Convolutional Neural Networks for Page Segmentation of Historical Document Images
• LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
• Multiple Document Datasets Pre-training Improves Text Line Detection With Deep Neural Networks
• Robust Text Line Detection in Historical Documents: Learning and Evaluation Methods
• DocBank: A Benchmark Dataset for Document Layout Analysis
• Chargrid: Towards Understanding 2D Documents
• Abstractive Information Extraction from Scanned Invoices (AIESI) using End-to-end Sequential Approach
• StructuralLM: Structural Pre-training for Form Understanding
• Extracting Variable-Depth Logical Document Hierarchy from Long Documents: Method, Evaluation, and Application
• Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learning
• SelfDoc: Self-Supervised Document Representation Learning
• LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding
• Efficient Document Image Classification Using Region-Based Graph Neural Network
• Cross-Domain Document Object Detection: Benchmark Suite and Method
• VTLayout: Fusion of Visual and Text Features for Document Layout Analysis
• Fine-Grained Object Detection over Scientific Document Images with Region Embeddings
• Human-In-The-Loop Document Layout Analysis
• Accurate Fine-grained Layout Analysis for the Historical Tibetan Document Based on the Instance Segmentation
• Form2Seq : A Framework for Higher-Order Form Structure Extraction
• DeepErase: Weakly Supervised Ink Artifact Removal in Document Text Images