Robust Layout-aware IE for Visually Rich Documents with Pre-trained Language Models
arXiv:2005.11017
Abstract
Many business documents processed in modern NLP and IR pipelines are visually rich: in addition to text, their semantics can also be captured by visual traits such as layout, format, and fonts. We study the problem of information extraction from visually rich documents (VRDs) and present a model that combines the power of large pre-trained language models and graph neural networks to efficiently encode both textual and visual information in business documents. We further introduce new fine-tuning objectives to improve in-domain unsupervised fine-tuning to better utilize large amount of unlabeled in-domain data. We experiment on real world invoice and resume data sets and show that the proposed method outperforms strong text-based RoBERTa baselines by 6.3% absolute F1 on invoices and 4.7% absolute F1 on resumes. When evaluated in a few-shot setting, our method requires up to 30x less annotation data than the baseline to achieve the same level of performance at ~90% F1.
arXiv comment10 pages, to appear in SIGIR 2020 Industry Track
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Passage Re-ranking with BERT
- DocBERT: BERT for Document Classification
- GraphIE: A Graph-Based Framework for Information Extraction
- Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
- Diverse Few-Shot Text Classification with Multiple Metrics
- Induction Networks for Few-Shot Text Classification
- Chargrid: Towards Understanding 2D Documents
- Robust Named Entity Recognition with Truecasing Pretraining