LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
arXiv:2104.08836
Abstract
Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually-rich document understanding tasks recently, which demonstrates the great potential for joint learning across different modalities. In this paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually-rich document understanding. To accurately evaluate LayoutXLM, we also introduce a multilingual form understanding benchmark dataset named XFUND, which includes form understanding samples in 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese), and key-value pairs are manually labeled for each language. Experiment results show that the LayoutXLM model has significantly outperformed the existing SOTA cross-lingual pre-trained models on the XFUND dataset. The pre-trained LayoutXLM model and the XFUND dataset are publicly available at https://aka.ms/layoutxlm.
arXiv commentWork in progress
References in corpus (8)
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Multilingual Denoising Pre-training for Neural Machine Translation
- mT5: A massively multilingual pre-trained text-to-text transformer
- Joint entity recognition and relation extraction as a multi-head selection problem
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
- LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
- DocVQA: A Dataset for VQA on Document Images