Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learning

Abstract

Recent approaches in literature have exploited the multi-modal information in documents (text, layout, image) to serve specific downstream document tasks. However, they are limited by their - (i) inability to learn cross-modal representations across text, layout and image dimensions for documents and (ii) inability to process multi-page documents. Pre-training techniques have been shown in Natural Language Processing (NLP) domain to learn generic textual representations from large unlabelled datasets, applicable to various downstream NLP tasks. In this paper, we propose a multi-task learning-based framework that utilizes a combination of self-supervised and supervised pre-training tasks to learn a generic document representation applicable to various downstream document tasks. Specifically, we introduce Document Topic Modelling and Document Shuffle Prediction as novel pre-training tasks to learn rich image representations along with the text and layout representations for documents. We utilize the Longformer network architecture as the backbone to encode the multi-modal information from multi-page documents in an end-to-end fashion. We showcase the applicability of our pre-training framework on a variety of different real-world document tasks such as document classification, document information extraction, and document retrieval. We evaluate our framework on different standard document datasets and conduct exhaustive experiments to compare performance against various ablations of our framework and state-of-the-art baselines.

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Overview of Multi-Task Learning in Deep Neural Networks
• Longformer: The Long-Document Transformer
• Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network
• CUTIE: Learning to Understand Documents with Convolutional Universal Text Information Extractor
• Modular Multimodal Architecture for Document Classification

Cited by in corpus (5)

• A Survey of Deep Learning Approaches for OCR and Document Understanding
• LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding
• SelfDoc: Self-Supervised Document Representation Learning
• StrucTexT: Structured Text Understanding with Multi-Modal Transformers
• ViBERTgrid: A Jointly Trained Multi-Modal 2D Document Representation for Key Information Extraction from Documents